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(12) "Program Review and Unit-Level Effectiveness Assessment: Heating, Air 
Conditioning, and Refrigeration Technology (HART)"; (13) "ACC Student 
Engagement ... Educational Bliss?" and (14) "Transfer Outcomes: Start 
Here. . .Get There and Succeed." (AA) 
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The ''Fast TracA’*... Students Say It Works! 



Many community colleges aroimd the coimtry 
have begim to offer courses in non-traditional in- 
structional formats in response to their community’s 
call for increased access to educational opportimi- 
ties. In the fall of 1998, ACC offered 
courses in an eight-week semester 
format in twenty-four disciplines, 
from art to welding, and including 
history, government, foreign lan- 
guage, English, math, computer sci- 
ence, and developmental studies. 

The coursework of these classes was 
“accelerated.” Class meetings were 
altered so students could earn the 
same number of credit hours as in a 
16-week semester course. 

Students in eight week semester 
classes were surveyed to provide fac- 
ulty and administrators information to 
insure continued quality educational 
experiences for students. 

This article presents a summary 
of student "feedback" from that sur- 
vey. The complete Eight Week 
Semester Course Survey Report is 
available from the Office of Institutional Effective- 
ness and on the ACCweb at the following address: 
http://accweb.austin.cc.tx.us/oie. 

METHODOLOGY 

Survey Administration: Two questionnaires 

were designed to collect student feedback data on 
8-week semester courses, a Before 8-week Course 
Feedback questionnaire to be completed during the 
first week of the class and an After 8-week Course 
Feedback questionnaire to be completed during the 
last week of the class. Faculty distributed question- 
O 
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Who enrolled in the 
8-week semester courses? 

Why did they enroll in this 
course format? 

What were their 
expectations and 
experiences of the 
8-week course? 

How did they do? 



naires to their students to complete during class time 
and returned the completed questionnaires to the 
Office of Institutional Effectiveness for analysis. 

Instrument: Each questionnaire consisted of a 
set of 14 “reasons” students might 
select for taking the 8 -week course, 
a set of demographic data items, 
and a set of statements describing 
students’ “expectations” {Before 
8-week questionnaire) or 
“experience” {After 8-week ques- 
tionnaire) of the course, with 
which students indicated their level 
of agreement (strongly agree, 
somewhat agree, somewhat dis- 
agree, and strongly disagree). 

Survey Population: The sur- 
vey population consisted of 39 
(49.4 percent) of the 79 course 
sections offered in the first 8-week 
term of Fall 1998. At least one 
section of each course, every cam- 
pus where an 8-week course was 
taught, every meeting time 
(morning, middle of the day, and 
evening), and every meeting format (twice/week, 
four times/week, once/week, three times/week, five 
times/week, Saturdays, and week days) were repre- 
sented in the 39 sections selected. 

(Continued on page 2) 
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(Continued from page 1) 

RESULTS 

Twenty-two of the 39 selected sections returned 
both the Before and After %-Week Course Feed- 
back questionnaires, an overall return rate of 56.4 
percent. The return fate for Before questionnaires 
only was 61.5 percent (24 of 39 sections, 416 
students). The return rate for After questionnaires 
only was 91.7 percent (22 of the 24 sections, 272 
students). After questionnaires were sent only to 
those sections that had completed the Before 8- 
Week Course Feedback questiormaire. Analysis 
of the Before and After questiormaires revealed 
significant similarity of response on both ques- 
tiormaires. Thus, in the report that follows, per- 
centages are given for responses by students who 
completed the 8-week course, except in the case 
of questions relating to student’s expectations 
where percentages are given for responses by 
students who completed the Before questionniare. 

Who enrolled in the 8-week courses? 

Based on the responses to both the Before and 
After questiormaires, the greatest percentage of the 
students who enrolled in an 8 -week semester 
course in Fall 1 998 

• Were between the ages of 1 7 - 2 1 years old 
(43.0%) 

• Had earned 36 or more credit hours to date 
from ACC and any other college (34.6%) 

• Attended classes at ACC primarily in the 
daytime (67.8%) 

• Were employed part or full time (43 .0%) 

• Were enrolled in 12 - 14 credit hours in 
Fall 1998, including the 8-week semester 
course (30%) 

• And were NOT concurrently attending an- 
other college (89.9%). 

Why did they enroll in this course format? 

A majority of students cited the following 
reasons for taking a class in the 8-week semester 
format (listed in descending order of number and 
percent of students selecting that reason): 



• I can take more courses in a single semester 
(one in the first 8 -weeks, another in the next 
8-weeks). (N= 189, 69.5%) 

• I can complete a two course sequence in one 
semester (i.e., ENG 1613 the first 8-weeks, 
ENG 1623 the second). (N=166, 61.0%) 

• The 8 -week course fits into my life/work 
schedule better than long semester courses 
do. (N= 165, 60.7%) 

Other of the top ten reasons for taking a class in 
the 8-wepk semester included the following : 

• I prefer the accelerated pace of the 8-week 
course. (N=134, 49.3%) 

• I can take a course that I am not really 
interested in, but that is required for my 
degree, and "get it over" quickly. (N= 113, 
41.5%) 

• I tend to “lose steam” as a semester pro- 
gresses. Knowing the class ends in 8 weeks 
takes the pressure off. (N=98, 36.0%) 

• I learn better when I can focus on fewer 
courses at a time. (N= 92, 33.8%) 

• I can start a course in the middle of the 
semester. (N= 92, 33.8%) 

• I can meet the requirements for develop- 
mental courses sooner. (N= 87, 32.0%) 

• I am likely to procrastinate if assignments 
are spaced at time intervals that are too 
wide. (N=87, 32.0%) 

Expectations... 

The majority of students who completed the 
Before 8-week Course questiormaire expected the 
workload would be modified (85.8%), but not 
"watered down." (73.8%) They expected to learn 
as much (88.7%) and spend as much time in 
preparation (79.5%) for the 8 -week course as for 
16-week courses. They expected (80.8%), and 
were willing (85.8%), to work twice as hard in the 
8-week course as in a 16-week course. 

Most intended to take another course in the 
second 8-week semester (79.3%) and wanted ACC 
to offer more 8-week courses (88.3%). 

(Continued on page 3) 
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...and Experience 

The 8-week courses met students’ expectations 
expressed on the Before 8-week questionnaire. 
The majority of students completing the After 8- 
week questionnaire indicated the workload and 
course content were manageable (83.8%), but not 
"watered down" (85.3%); they learned as much in 
the 8-week course (73.1%) and would not have 
made a higher grade in the 16-week version of the 
same course (59.6%). Students reported they spent 
as much time in preparation for the 8-week course 
as they would have in a 16-week course (67.3%). 
Preparation for the 8 -week course took more time 
than they had expected it would (58.5%). They 
worked twice as hard in the 8-week course (66.9%) 
and felt it was worth it to complete courses, or their 
degree, earlier than in a 16-week semester (82.4%). 

The majority of students completing the After 
8-week questionnaire (the students who completed 
the 8-week course) intended to take another course 
in the second 8-week semester (69.9%), or future 
8-week semesters (58.9%), and they would recom- 
mend 8-week courses to other students (89.7%). 
They wanted ACC to offer more 8-week courses 
(90.1%). 

The After questionnaire requested that students 
list three advantages and three disadvantages of the 
format. Predominate “advantage” themes in these 
comments were about the accelerated pace of the 
class, no time to become bored, focus of the short 
time compared to the “long” semester, completing 
degree requirements quickly. Predominate 
“disadvantage” themes in these comments were 
about the amount of work involved, the time re- 
quired to keep up with the workload, and the 
danger of falling behind — ^there is no catch up time 
available. Also mentioned were the compressed 
exam schedule (less time between exams), the ac- 
celerated pace of the course, and concern whether 
learning would be retained. Some students did not 
like having to spend long hours in class with the 
same people. Some felt the 8-week course took 
time from their 1 6-week course preparation. 
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How did the students do in the 8-week courses? 

Students in the 8-week semester courses did 
very well. The grade distribution for 338 of the 
416 students participating in the feedback survey 
(two sections did not have grades when the data 
were run) were as follows: 

Grade Distribution for Fall 1998 8-Week Courses 



Grade 


N 


% 


A 


109 


32.2% 


B 


99 


29.3% 


C 


46 


13.6% 


D 


7 


2.1% 


F 


19 


5.6% 


W 


46 


13.6% 


1 


11 


3.3% 


AU 


2 


0.6% 



In the sections from which completed surveys 
were received, 254 of 338 students (75.1%) made 
a grade of A, B, or C. Students taking the 8-week 
semester courses persist in these courses at a 
much greater rate than in some other course for- 
mats. Compared to withdrawal rates for .other 
course formats, the withdrawal rate for the 8-week 
semester courses (13.6 %) was below the college- 
wide average withdrawal rate of 20.5 percent . 

CONCLUSION 

The results of this study indicate 8-week for- 
mat courses meet students’ needs and the majority 
of students who take them do well. The majority 
of students completing 8-week semester courses 
say they will take another course in this format — 
either in the next 8-weeks or at another time. 
They ask that more of the 8-week courses be 
offered. 

— Roslyn Wallace 
Institutional Studies Specialist 

OIE 

The complete study report is available at 
http://accweb.austin.cc.tx.us/oie 
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The President s Effectiveness council (PEC) is an advisory body, one of whose functions is to discuss and 
analyze data and information relevant to institutional effectiveness and to refer the results of these discussions to 
other councils, committees, and task forces for their attention and action, to promote improvement of institutional 
quality. At its November, 1998 meeting, PEC reviewed a study, "Course Completions Analysis of Distance Learning 
Courses. Ron Brey, AVP Open Campus, reported Open Campus (OPC) goals and strategies for increasing student 
success in those courses. The first article that follows is a summary of the study. The second article is OPC’s 
improvement plan. 

The complete study is available from the OIE or online at http://accweb.austin.CC.tx.us/oie. 



Are Students Successfully Completing Distance Learning Courses at ACC? 



One of the primary functions of a commu- 
nity college is to ensure the successful completion 
of coursework by its students. Originally, a study 
was undertaken by the Office of Institutional Ef- 
fectiveness (OIE) to compare rates of completion 
and passing grades for credit courses at ACC. 
This study is a supplement to that analysis and 
shows completion and passing grades for Distance 
Learning (OPC) courses at ACC. Six types of 
OPC exist at ACC: 

• Telecourses (ITV) using recorded video pro- 
grams 

• Live televised classes (LTV) originating at the 
Pinnacle Campus in Southwest Austin 

• Email and Internet courses (PC-Based) 

• Print-Based courses that are textbook-based 

• Directed study courses 

• VTEL courses, consisting of the Vocational 
Nursing program with Fredericksburg 

Analysis was done both by OPC type and 
in comparison to classroom-based counterparts to 
establish the relative success rates of the OPC 
program at ACC. 

Methodology 

Data analyzed was from the Fall 1997 
semester. Course D-F rates were defined as a ratio 
of grades of D or F over all possible grades of A, 
B, C, D, F, or W. Course withdrawal rates were 
defined as a ratio of grades of W over all possible 
grades. The sum of the D-F rate and the with- 
drawal rate is the D-F-W rate. This rate implies 
failure to successfully complete the course. Stu- 




dent grades are counted once for each course 
taken.- 

Analysis was conducted on OPC type, de- 
partment, course, and student demographics. 
These were compared to the OPC average rates of 
D"F grades and withdrawal. Only those depart- 
ments with enrollments of more than 100 students 
were included. Rates for OPC departments and 
courses were also compared to their classroom 
counterparts. 

Additional analysis was conducted on. 120 
student subpopulations that were created using 
age, ethnicity, and gender, and these subpopula- 
tions were compared to the averages for OPC and 
also to classroom averages. Any subpopulation 
with a D-F grade rate or withdrawal rate more than 
ten percentage points above the OPC average was 
considered noteworthy. Only those subpopula- 
tions with enrollments of greater than 20 students 
were considered because with lower enrollments, 
only a few D-F grades or withdrawals could skew 
the total percentage. 

Results 

For OPC, the average D-F grade rate was 
14.7% and the average rate of withdrawal was 
32.9%, for a total D-F-W rate of 47.6%. This was 
substantially higher than the rates for classroom- 
based learning, where the average D-F grade rate 
was 9.0% and the average rate of withdrawal was 
20.5%, for a total D-F-W rate of 29.5%. 

The OPC departments shown in Table 1 
had D-F-W rates above the OPC average of 47.6%. 
Of these departments. Accounting, English, and 

(Continued on page 5) 
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Table 1 

OPC Departments with D-F-W Rates Above OPC Average 



Department 


OPC Type 


OPC 

Enrollment 


OPC 

D-F-W Rate 


Classroom 
D-F-W Rate 


Accounting 


ITV 


116 


72.4% 


30.5% 


Anthropology 


ITV, Print-Based 


127 


57.5% 


20.9% 


Sociology 


ITV, Print-Based, Direct 


254 


55.1% 


28.6% 


English 


ITV, PC-Based 


307 


52.4% 


34.2% 


Philosophy 


ITV 


108 


51.9% 


32.1% 


Government 


ITV, Print-Based 


472 


50.4% 


27.1% 


Average 






47.6% 


29.5% 



Philosophy had above average D-F-W rates for both classroom-based learning and OPC environments. 

Table 2 shows the OPC departments with withdrawal rates above the OPC average of 32.9%. Of these 
departments, Accounting, English, and Economics had above average rates of withdrawal for both 



Table 2 

OPC Departments with Withdrawal Rate Above OPC Average 



Department 


OPC Type 


OPC 

Enrollment 


OPC 

WRate 


Classroom 

WRate 


Accounting 


ITV 


116 


52.6% 


21.1% 


Philosophy 


ITV 


108 


44.4% 


19.3% 


English 


ITV, PC-Based 


307 


40.7% 


25.9% 


Sociology 


ITV, Print-Based, Direct 


254 


39.0% 


19.2% 


Psychology 


ITV, Print-Based 


264 


36.0% 


16.4% . 


Economics 


ITV 


192 


35.9% 


23.0% 


Allied Health Sciences 


LTV, Print-Based 


113 


33.6% 


15.2% 


Computer Info. Systems 


PC-Based 


328 


33.2% 


20.2% 


Anthropology 


ITV, Print-Based 


127 


33.1% 


14.3% 


Average 






32.9% 


20.5% 



classroom-based learning and OPC environments. 

Next, OPC departments were compared to 
their classroom-based counterparts. The D-F-W rate 
by department was greater for OPC than for 
classroom-based learning in all departments except 
Physical Science and Vocational Nursing. Addition- 
ally, the percent of enrolled students v^thdrav^ng in 
a department was greater for OPC than for 
classroom-based learning in all departments except 
Biology, Physical Science, and Vocational Nursing. 

OPC types were compared to establish which 
OPC types contributed the largest number of D-F 
grades and withdrawals. These numbers were 
closely correlated with percent of enrollment; how- 
ever, ITV learning contained slightly more than its 

O 
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share of all three categories. ITV had 57% of the 
total OPC enrollments, but 62% of the total OPC 
withdrawals, D-F grades, and D-F-W. 

Men, Blacks, Hispanics, and all age groups 
between 18 and 24 had above average rates in all 
three areas analyzed: withdrawal, D-F grades, and 
D-F-W. This was no different than classroom- 
based learning, where Men, Blacks, and Hispanics 
were above average in all three categories as well. 

The following student subpopulations had 
D-F-W rates more than ten percentage points above 
average: 

(Continued on page 6) 
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(Continued from page 5) 

Black Females age 20-2 1 82.1% 

Hispanic Males age 20-21 75.0% 

White Males age 20-21 64.5% 

Hispanic Females age 20-21 64.3% 

White Males age 18-19 62. 1 % 

Asian Females age 25-29 59. 1% 

Asian Males age 20-21 59.1% 

White Males age 22-24 59.1% 



Black Females age 20-21 appeared in all three 
sections (withdrawals, D-F grades and D-F-W). 
Black Females age 20-21 appeared in two of the 
three sections for classroom-based learning as well. 

Conclusion 

• Almost half of all OPC enrollments ended in 
grades of D or F or in withdrawal, substantially 
higher than the rates for classroom-based learn- 
ing. 

• Men, Hispanics, Blacks, and younger students 
(those under 25 years of age) had greater diffi- 
culty than did the average student at succeeding 
in OPC courses. However, these same groups 
also had greater difficulty than average at suc- 
ceeding in traditional classroom courses. 

While this study shows a greater overall proba- 
bility of succeeding in a classroom-based course 
than in OPC, student responses to OPC were not 
considered. Students may appreciate the conve- 
nience and other advantages of OPC enough to 
outweigh the higher than normal D-F-W rate in- 
volved with these courses. OPC is certainly a part 
of the future in higher education. Having specific 
knowledge about its positive and negative outcomes 
at ACC will help us to best utilize OPC as an 
instructional technique. 

— Vickt Retd 
Intern 
OIE 



Distance Learning... 

Increasing Course Completions 

OPC Course Completions Goals 

• Goal 2000: Reduce the difference between the 
average D-F-W rate for on-campus and distance 
education students from 18.1 percent to 10 per- 
cent. 

• Goal 2003: Reduce the difference between the 
average D-F-W rate for on-campus and distance 
education students from 18.1 percent to no more 
than 8 percent. 

Data presented in the OIE study of OPC course 
completions support the historical nationwide 
trends in OPC courses. 

• Successful completion rates are significantly 
lower than the on-campus equivalent with most 
of the difference being explained in withdrawal 
rates. A goal held by some community col- 
leges is to have the difference [between OPC 
courses and classroom based course comple- 
tion rate] no greater than 10 percent. 

® There is a direct correlation between the age of 
the distant learning student and the probability 
of success. 

• There are significant variations among courses 
and, sometimes, sections of the same course. 
Variations are sometimes explainable by dif- 
ferences in course syllabi, but often there is no 
explanation. 

Over the past 20 years, ACC has collected and 
analyzed data on the success rates of students in its 
OPC program. These data support those in the 
study and lead to additional conclusions: 

• Motivation and time management skills are the 
most important personal skills for success in a 
distance education course; thus the correlation 
of success with age. 

• Summer success rates in distance education 
courses are significantly higher due to the differ- 
ences in the background of the average student. 

(Continued on page 7) 
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Summer students tend to be successfully at- 
tending another college or university, are 
highly motivated and have better skills. 

• Evaluation of reported reasons for withdraw- 
ing indicate that the large majority are non- 
college related; e.g., college conflicts with 
work, enrolled in too many courses, and fam- 
ily commitments. 

• Students who succeed in distance education 
courses rate them positively. 

Actions to Reduce D-F-W Grades for Open 
Campus Students 

Intervention strategies to improve student 
completion rates can occur in three general time 
periods. Each period has a different set of objec- 
tives and strategies. 

1. Before Enrolling 

These strategies will be instituted for the Fall 
1999 semester. 

• Have the student complete an on-line student 
assessment that will be a high predictor of 
their success in the course. The student will 
receive positive feedback on his/her strengths 
and recommendations to address weaknesses. 

• Revise the Open Campus information in the 
Course Schedule to improve student under- 
standing of what it takes to succeed in a 
distance education course. 

• Provide better information to faculty, coun- 
selors, and staff about Open Campus courses. 

• Produce a video program on Open Campus, 
the types of courses offered and what it takes 
to succeed in a distance education course. 
This will be aired on ACC’s cable television 
channel. 

• Post the syllabus for each Open Campus 
course on the OPC web site so that students 
will make better informed decisions before 
enrolling. 
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2. After Enrolling and Before Classes Start 

This is an important time period when stu- 
dents should receive accurate information about 
their course orientation, instructional materials 
and activities. Also, if students decide that they 
may not have made a good decision, it is still 
possible to change to an on-campus class. 

• Mail to each student registered for an Open 
Cajnpus course a customized letter within sev- 
eral days of their enrolling. This letter will 
contain information about the course and gen- 
eral information on how to succeed in a dis- 
tance education course. If the student feels 
they made an inappropriate decision to take an 
OPC course, there is time to drop the course 
and add an on-campus course. 

• Offer a workshop on basic skills in succeeding 
in distance education courses. 

3. During the Semester 

Some courses and students would benefit 
from more periodic contact between faculty and 
Open Campus during the session. This would 
help to pace students through the session and 
provide better feedback on their progress. 

• Set a minimum standard for the number of 
contacts between a faculty member and stu- 
dents. 

• Provide computerized academic testing which 
will give students immediate detailed feed- 
back when they take an exam at the testing 
center. 

• Review each course syllabus for clarity, con- 
sistency with departmental guidelines, the ef- 
fectiveness of instructional materials and 
course activities. 

— Ron Brey 
AVP 
OPC 
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Q: I know departments are designing assessment 

plans for determining whether their program goals are 
being met, but what about collegewide goals like 
access, transfer, retention and worlforce preparation? 
How do we know whether these ’’missions” of the 
College are being achieved? 




specific expertise to improve effectiveness 
(outcomes achievement) of the College’s missions. 

Q: Where is PEC in this process? 



A: As with program goals, collegewide goals must be 
assessed for the College to determine whether its 
“missions” are being achieved. The steps involved in 
designing assessment plans for collegewide goals are 
the same, however, as for program goals: 

1. decide the specific outcomes to be achieved 

2. generate and clarify questions that must be 
answered to demonstrate achievement of these 
outcomes 

3. define acceptable levels of outcomes achieve- 
ment (standards) 

4. identify and gather data necessary to answer 
outcomes achievement questions 

5. propose improvement actions 

Q: Who is responsible for designing collegewide 

assessment plans? 

A: Unlike program-specific goals, collegewide goals 
(missions) involve all disciplines and units of the 
College. Thus, designing assessment plans for col- 
legewide goals is the responsibility of a collegewide 
group. At ACC, this group is the President’s Effec- 
tiveness Council (PEC, see p. 4). 

PEC is composed of faculty, deans, AVPs, 
provosts, and administrators, each contributing area- 



A: In Spring 1997, when the PEC was created, it 
began to address effectiveness of the transfer mis- 
sion. Over a period of several meetings, the Coun- 
cil suggested, discussed, and refined questions that 
must be answered to determine whether ACC is 
achieving its transfer mission. The Council agreed 
on standards for defining “achievement” of the 
mission and identified data to be used. By Fall 
1997, the assessment plan for transfer was com- 
pleted. 

The same process was used during academic 
year 1997-1998 to design assessment plans for stu- 
dent access, student retention and workforce prepa- 
ration. 

Assessment is an ongoing process, continually 
refining outcomes questions and identifying data to 
answer those questions. This academic year, PEC is 
examining data for each mission and proposing 
refinements to the assessment plans where needed. 

The Effectiveness Update reports the status of 
each mission. Beginning this month, it will be 
posted on the OIE web site 

(http://accweb.austin.cc.tx.us/oie), 
as well as distributed via intercampus mail. Watch 
for it in your mailbox. 
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Tracking Transfers 



In April, 1995, the Texas Association of Junior and 
Community College Instructional Administrators 
(TAJCCIA) recommended Texas community colleges, 
"collect and maintain data to identify specific problems 
and improve results of student transfer from two-year to 
four-year colleges." {Transfer Success Work Group 
Report, April 1995.) 

In response to that recommendation, the President’s 
Effectiveness Council (PEC) examined data to answer 
the following questions relevant to ACC's transfer func- 
tion: 

• At what rate do ACC students transfer to senior 
institutions? 

• To which senior institutions do they go? 

This article presents a condensed version of the PEC 
report. The complete report is available from the 
Office of Institutional Effectiveness or on the OIE web 
site. 

At what rate do ACC students transfer to universities? 

The THECB Transfer Effectiveness Report pro- 
vides transfer data for cohorts of first-time-in-college 
(FTIC) students who enrolled in summer or fall of a 
given year, attended at least two semesters, and earned 
at least 15 semester credit hours (SCH). Four years 
following the first enrollment of the cohort, a transfer 
rate is calculated by the THECB that compares the 
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number of students in the cohort who are enrolled in 
a Texas public senior institution to the number of 
students in the cohort who are no longer enrolled in 
the community college. Cohort students still en- 
rolled in a two-year institution at the time of the 
measure are removed from calculation of the trans- 
fer rate. 

This rate answers the question, "What percent 
of students who began their college education at a 
community college and are no longer enrolled in 
that community college, have continued their educa- 
tion at a senior institution within four years of their 
first community college enrollment?" This rate is 
the one commonly referred to when speaking of a 
community college's "transfer rate." Table 1 dis- 
plays ACC and statewide FTIC transfer rate data 
from the Transfer Effectiveness Report for four 



Cohort Year 


FTIC 

Students 


Students with at 
least 15 SCH in 2 
Semesters 


Students 
Not Still 
Enrolled 


Transfer 

Students 


Transfer 

Rate 


ACC 1993 


3,787 


2,541 


1.649 


495 


30.0% 


State 1993 


82,752 


56,904 


39,640 


10,474 


26.9% 


ACC 1992 


3,852 


2,592 


1,645 


446 


27.1% 


State 1992 


83,277 


57,552 


13,358 


10.308 


26.0% 


ACC 1991 


3,723 


2,506 


1,650 


528 


32.0% 


State 1991 


82,076 


56,071 


38,458 


9.880 


25.7% 


ACC 1990 


3,801 


2,424 


1,495 


476 


31.8% 


State 1990 


77,144 


51,453 


34,790 


8,812 


25.3% 



Source: Transfer Effectiveness Reports. 1990- 1993 



FTIC cohort years, 1990-1993. 

ACC FTIC transfer rates compare favorably 
with statewide FTIC transfer rates for each of the 
four cohort years displayed. For each cohort year, 
of the FTIC cohort students who did not return to 

(Continued on page 2) 
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ACC, nearly one third were enrolled in a Texas public 
senior institution within four years following their first 
enrollment at ACC. For this same time period, one 
fourth of all statewide FTIC cohort students were 
enrolled in a Texas public senior institution. 

It is worth noting, however, that the percent of 
change from one year to the next for ACC’s FTIC 
transfer rates varies much more than the percent of 
change for statewide transfer rates. Whereas the 
statewide change in FTIC transfer rate from one year 
to the next has varied slightly in a downward trend, 
ACC's transfer rate fluctuates from one cohort year to 
the next, and for the most current cohort year, is higher 
than the previous year. 

To which universities do they transfer? 

Identifying the senior institutions in which 
students enroll when they leave ACC assists the Col- 
lege in inviting articulation agreements with, and re- 
questing performance data from, those senior institu- 
tions, thereby improving the transfer process for stu- 
dents. The THECB report. Students Pursuing Addi- 
tional Education— By Institution (Graduates and Non- 
Returners), tracks non-returning students who were 
enrolled in a Texas public community or technical 
college during a given academic year and then en- 
rolled the following fall semester in any Texas public 
higher education institution. For each community or 
technical college, the report lists the number of its 
non-returning (including graduates) students who 
transferred to community colleges, universities, or 
health sciences centers. Beginning with the 1996 aca- 



demic year, the THECB provided a statewide compari- 
son report of these data. Data from this report can be 
used to answer the question, "Of the students who 
transfer to another institution during the fall semester 
after leaving ACC, what proportion enrolled in Texas 
public universities?" 

The Students Pursuing Additional Education— 
By Institution (Graduates and Non-Returners) report 
also provides information on which specific institu- 
tions ACC graduates and non-returning students trans- 
ferred to from one academic year to the next. Data for 
the past five years indicate approximately 90 percent 
of ACC's non-returning students who transferred to 
Texas public universities enrolled in one of four uni- 
versities to continue their education: The University of 
Texas at Austin (UT), Southwest Texas State Univer- 
sity (SWTSU), Texas A&M University (TAMU), or 
Texas Tech University (TTU). Those who do not 
transfer to these universities enroll in one of 3 1 other 
Texas public universities. Table 2 displays these data. 

Consistently over the past five years, just un- 
der two thirds (63.4% - 64.3%) of ACC's non- 
returning students who transferred to a Texas public 
university enrolled in the University of Texas at 
Austin following their last ACC enrollment, and about 
one fifth (19.1% - 21.3%) enrolled in Southwest Texas 
State University to continue their education. The 
percent of students transferring to SWTSU has been 
increasing steadily since academic year 1992-93, 
while the percentages of students enrolling in UT and 
all other Texas public universities has fluctuated. 

— President’s Effectiveness Council 



Year 


Table 2: Texas 

University of Texas 
at Austin 


Public Universities tc 

Southwest Texas State 
University 


) which ACC's Non- 

Texas A&M 
University 


Returning Studen 

Texas Tech 
University 


ts Transferred 

31 Other Texas Public 
Universities 


Total 




N 


% 


N 


% 


N 


% 


N 


% 


N 


% 


N 


1996-97 


5.057 


63.6% 


1,691 


21.3% 


340 


4.3% 


162 


2.0% 


701 


8.8% 


7,951 


1995-96 


5,697 


63.8% 


1,844 


20.6% 


366 


4.1% 


201 


2.2% 


828 


9,3% 


8.936 ^ 


1994-95 


5.512 


63.4% 


1.754 


20.2% 


439 


5.0% 


196 


2.3% 


796 


9,2% 


8.697 


1993-94 


5.393 


64.3% 


1,641 


19.6% 


406 


4.8% 


183 


2.2% 


770 


9.2% 


8.393 ^ 


1992-93 


5.270 


64.2% 


1.565 


19.1% 


441 


5.4% 


134 


1.6% 


794 


9.7% 


8.204 : 



Source: Academic Year 1 996-97 Students Pursuing Additional Education-By Institution (Graduates and Non-Returners) 
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Graduates Respond — Favorably! 

Each fall, Austin Community College surveys its 
graduates of the previous year to collect information 
regarding the effectiveness of both transfer and work- 
force programs. Additionally, the Texas Higher Educa- 
tion Coordinating Board requires colleges to annually 
submit outcomes data to determine program effective- 
ness and funding. 

The ACC 1996-97 Graduate Survey was distributed 
during the spring of 1998 to 1,089 graduates of the 
1996-97 academic year. A total of 505 (46.4%) re- 
sponses were received. Graduates were asked to re- 
spond to questions about their current employment sta- 
tus, whether their degree or certificate was required for 
their job, any further education beyond their ACC work, 
any volunteer work involving ACC training, and their 
opinion of how well ACC prepared them for employ- 
ment or further education. 

The chart below illustrates, of the 1,089 1996-97 
graduates, 447 graduates (41.0%) are (or were, at one 
time) employed. Of those, 372 (34.1%) were employed 
in a job that required training in the field in which they 
majored. 



Additionally, of the graduates responding to the 
survey, most rated ACC’s preparation for employ- 
ment as “excellent” (53.3%) or “satisfactory” 
(29.5%). 



Please rate how well you feel ACC prepared 
you for employment or further education. My 
preparation was... 




N 


% 


Excellent 


269 


53.3 


Satisfactory 


149 


29.5 


Good only in some areas 


59 


11.7 


Fair 


20 


4.0 


Inadequate 


2 


0.4 



— Joe Vasquez 
Research Specialist 
Institutional EfFectiveness 



Graduate Follow-Up Survey Results 1996-97 Graduates 

(Note: all percentages are of the total number of 1996-97 graduates.) 
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An Orientation Toward Success 

As part of the Human Development program’s as- 
sessment process for 1998/99, we decided to assess the 
effectiveness of HDP 1601. We gathered qualitative 
data with feedback forms collected from students who 
completed the course. With the assistance of the Office 
of Institutional Effectiveness, we designed a study to 
determine whether our intended outcomes of the course 
were achieved. This article summarizes our quantita- 
tive assessment study. 

Purpose and Intended Outcomes 

HDP 1601, a new Orientation course developed 
during the spring and summer, and implemented during 
the fall of 1998, was designed to increase the success 
and retention of incoming freshmen students who failed 
either two or more skill areas of the assessment tests or 
one area at a fundamental level, i.e., ”at risk” students. 
The course would facilitate students' transition into 
Austin Community College and provide them a strong 
foundation for effective learning, thereby increasing the 
likelihood that they would continue in college. Objec- 
tives of the course were the following: 

• to provide students an orientation to the college 
experience, ACC, and the course itself 

• to assess students’ current expectations, motiva- 
tions, strengths and areas for improvement 

• to assist students in setting effective academic, 
personal, and career goals 

• to familiarize students with college support ser- 
vices and resources 

• to develop and increase students' self-management 
skills for creating successful outcomes 

• to provide students an introduction to essential 
learning and study skills for academic success. 

Assessment of Outcomes 
In Fall 1998 there were 7,707 entering students. Of 
these, 1,862 students failed two or more areas of the 
assessment test or failed at least one area at the funda- 
mental level. Student Services Counselors enrolled 342 
of these "at risk" students in the HDP 1601 Orientation 
course, while the remaining 1,520 students did not 
enroll in the course. Study cohorts were identified as 
follows: 

• Cohort I-Successful Completers: Those stu- 
dents who took the course and successfully com- 
pleted it with a grade of A, B, or C (N=262) 
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• Cohort Il-Un successful Completers: Those stu- 
dents who took the course but did not successfully 
complete it (grades of D, F, W, or I) (N=80) 

• Cohort III-Did Not Take: Those students who 
were identified as "at risk" by their assessment 
scores, but did not enroll in the orientation course 
(N= 1,520) 

• Cohort IV-Not ”At Risk”: All remaining enter- 
ing students (N=5,845) 

Cohorts I through III identify students who begin their 
college careers at risk for failure and must take remedial 
level courses in reading, writing, or math prior to begin- 
ning college-level credit courses. Students in Cohort IV, 
on the other hand, have demonstrated college-level skills 
competencies allowing them to take college-level credit 
courses. They are included for benchmarking only. 

Intended Outcomes 

The student outcomes we expected to achieve were 
that successfully completing this one credit-hour, five- 
week long course would increase retention of “at risk” 
students and their success in college. Retention was 
defined as enrollment in the following semester (Spring 
1999); success was defined by four indicators as follows: 

• Grade point average 

• Hours attempted 

• Hours earned 

• Course withdrawals 

Assessment Procedures and Criteria 

Using student level data from the student informa- 
tion system, retention into the following semester was 
calculated and cohort averages for the student success 
indicators were compared. We would know the course 
had achieved the retention outcome if a higher percent- 
age of Cohort I students enrolled the following semester 
than Cohorts II and III. The other outcome would be 
achieved to the degree that there was a positive differ- 
ence between the means for Cohort I and Cohorts II and 
III on each of the success indicators. 

Summary and Analysis of Data 

The data show that a greater percentage of Cohort I: 

(Continued on page 5) 
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Successful Completers re-enrolled the following semester (Spring 1999) than all other cohorts. On each of the 
success indicators, Cohort I also performed better than any of the other cohorts. The differences in performance 
were found to be statistically significant in each case. Particularly remarkable were results in the areas of course 
withdrawals during the semester and retention into the following semester. See Table 1. 



Table 1: Cohort Retention and Performance by Success Indicator 



Success Indicator 
Means 


1: Successful 
Completers 
N=262 


II: Unsuccessful 
Completers 
N=80 


III: Did Not Take 
N=1,520 


IV: NofAtRisk” 
N=5,845 


Retention Rates 


66.8% 


27.5% 


53.1% 


53.4% 


GPA 


2.77 


1.03 


2.61 


2.67 


Hours Attempted 


7.30 


3.41 


5.97 


6.39 


Hours Earned 


6.56 


1.86 


5.26 


5.66 


Course 


0.33 


2.00 


0.50 


0.54 



Withdrawals 



An additional set of calculations was completed to determine the effect of this course on retention of minority 
populations. The retention rate for minorities was improved or maintained as a result of successfully completing 
the course. There was a dramatic difference between the re-enrollment of Hispanic students completing the course 
successfully and Hispanic students in all other cohorts. See Table 2. 



Table 2: Students Enrolling in the Subsequent Semester by Ethnicity 



Ethnicity 


Cohort 1: 
Successful 
Completers 


Cohort II: 
Unsuccessful 
Completers 


Cohort III: 
Did Not Take 


Cohort IV: 
Not “At Risk” 


White 


94 


(53.7%) 


12 


(54.5%) 


526 


(65.2%) 


2,146 


(68.8%) 


Black 


17 


(9.7%) 


1 


(4.5%) 


61 


(7.6%) 


121 


(3.9%) 


Hispanic 


53 


(30.3%) 


5 


(22.7%) 


169 


(20.9%) 


582 


(18.7%) 


Other* 


11 


(6.3%) 


4 


(18.1%) 


51 


(6.3%) 


270 


(8.6%) 


Total** 


175 


(100%) 


22 


(99.8%**) 


807 


(100%) 


100 


(100%) 



*Includes Asian, American Indian, Other **Totals do not always equal 1 00% due to rounding. 

Plans for Improvement 

The study confirmed achievement of the outcomes we were assessing: Successful completion of HDP 

1601 — Orientation did increase retention and success of students entering ACC with skills levels that place them 
“at risk.” 

We will analyze the qualitative data collected from students to determine future course improvements. Also, 
at the end of the spring semester, we will track success and retention of those students who re-enrolled to 
determine whether the outcomes this study identified are maintained over time. In Fall 1999, more sections of the 
Orientation course will be offered at more locations and times. 

— ^Tobin Quereau 
Associate Professor 
Human Development 
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Math Pilot : Study Skills Attached Labs 



Background 

The study skills pilot project was conducted in the 
Fall 1998 semester, at the Rio Grande campus. One 
section each of Basic Math Skills and Elementary 
Algebra were combined with the respective labs. All 
students in the lecture sections were required to co- 
enroll in the related lab course. 

Curriculum in the lab course was based on two 
books by Paul Nolting: Winning at Math, 3nd edition 
(cost is about $25) and Math Skills Workbook (cost is 
about $13). The textbook has 11 chapters, covering 
topics from time management to test taking skills. The 
workbook corresponds to chapters from the text, but 
the material is shortened and organized so students can 
answer questions as they read the chapter. All content 
in both the text and workbook was specifically written 
for the math student. More details on the curriculum 
content of the text and workbook, as well as weekly 
journals submitted by the instructors can be found on 
the Math Task Force web page: 

http://www.austin.cc.tx.us/mparker/tf98-99/commrepts/pilot.htm 

The syllabus for the lab classes was designed to 
closely follow the syllabus in their respective courses. 
For example, students worked in groups to prepare for 
each test in the course and analyzed their tests after 
they had been graded. All chapters in the workbook 
were covered, as well as the chapters in the text that 
were not duplicated in the workbook. In addition, the 
syllabus included videos on math attitudes, test prepa- 
ration, and test analysis. 

The instructors for the lecture/lab combinations 
were Betsy Kreisle (Basic Math Skills) and Irene Doo 
(Elementary Algebra). In addition, Diane Kramer, 
counselor at Rio Grande, worked closely with the 
instructors in a variety of ways. She administered and 
interpreted learning style inventories to students and 
instructor, presented guest lectures on learning strate- 
gies and test anxiety, and co-presented on learning 
disabilities and attention deficit disorder. She also 
assisted students with analysis of error types on their 
exams and how to modify study skills to correct fre- 
quently occurring errors. She provided individual writ- 
ten feedback to students and group discussion on stu- 
dent's math autobiography assignment. Some students 
also sought out individual counseling. 



Results 

Impact on Grades: The following grade distribu- 
tions were submitted at the end of the semester for both 
the attached lab sections and the lecture-only sections 
of both courses with both instructors. 

Based on the final letter grades for the students at 
the end of the semester, it is clear that the Basic Math 
Skills students in the attached lab section showed 



Grade 




Basic Math Skills 




Elementary Algebra 






Attached Lab 


Lecture Only 


Attached Lab 


Lecture Only ^ 1 


A 


4 


33% 


2 


13% 


2 


11% 


2 


15% ? 


B 


3 


25% 


2 


13% 


4 


22% 


2 


15% 


C 


1 


8% 


3 


20% 


4 


22% 


4 


31% : 


D 


0 


0% 


2 


13% 


2 


11% 


0 


0% 1 


F 


2 


17% 


0 


0% 


1 


6% 


2 


15% ; 


W 


2 


17% 


6 


40% 


5 


28% 


3 


23% [ 


Total 


12 


100% 


15 


100% 


18 


100% 


13 100%! 



higher performance results compared to the lecture- 
only sections. The difference in the Elementary Alge- 
bra sections was not as pronounced. Informal feedback 
from the instructors and counselors also indicated that 
the study skills curriculum would be most effective at 
the lower level courses in Developmental Math. 

Impact on Affective Factors: Anecdotal feedback 
from students, instructors, and the counselor showed a 
huge impact on organization of homework study and 
test preparation, as well as students' self-esteem, under- 
standing of themselves as learners, and attitudes to- 
ward math. For example, the math autobiographies, as 
a whole, were very negative in tone and revealed 
anxiety, fear, shame, anger, and even psychological 
and physical abuse associated with prior math experi- 
ences. The autobiographies also discussed many in- 
stances of math avoidance resulting in lost educational 
and occupational opportunities. Most students re- 
ported that the study skills class was a positive learning 
experience; for many, the only one associated with 
math. In the counselor's opinion and not at all antici- 

(Continued on page 7) 
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(Continued from page 6) 

pated, the class became an actual "therapeutic" experi- 
ence, with many students reporting great improvement 
in their self-esteem and reduction in academic anxiety. 
Many of the students in the study skills pilot said they 
benefited from learning how to study math and highly 
recommended the inclusion of a study skills curriculum 
with the developmental math courses. In fact, most of 
the students in the Elementary Algebra class who had 
previously taken Basic Math Skills and/or Pre-algebra 
at ACC expressed their wish that they had learned math 
study skills in their first developmental math class at 
the College. 

Recommendations 

In the Spring 1999 semester, one section each of 
Basic Math skills and Pre-algebra was combined with 
the respective study skills lab. Unfortunately, low 
enrollments prior to the start of the semester resulted in 
both sections being cancelled. Causes of the low 
enrollment could have been poor placement of the 
sections in the schedule, inadequate information pro- 
vided to the advisors, and reluctance of students to 
enroll in and pay for an additional lab section. 

Given the reluctance of students to sign up for 
extra classes that they do not perceive as necessary to 
achieving their educational goals, the recommendation 
is to discontinue the offering of mandatory attached 
labs. 

Since the number of students participating in the 
pilot is small, these recommendations are tenuous and 
further study is advised. However, since Nolting's re- 
search and the results of the one semester study skills 
pilot at ACC have both shown that study skills are an 
important factor in the success of developmental math 
students, it is recommended that a limited version of 
the study skills curriculum be integrated into all devel- 
opmental math courses. In the interest of keeping the 
additional cost to the students at a minimum, the Math 
Study Skills Workbook (cost about $13) will be the 
only required text for the optional study skills curricu- 
lum. In addition, the textbook Winning at Math will be 
available in the reference section of the learning re- 
source centers at each campus, as will be accompany- 
ing video(s) and software. The syllabus and instructor 
notes to accompany the workbook will be provided in 
the math manual, and all instructors would have the 
option of incorporating the study skills curriculum into 
their existing syllabus. 



Additional recommendations include counselors 
college-wide to acquire skills in working with math 
specific affective and learning issues and to provide 
individual counseling and workshops for students tak- 
ing developmental math. It is also recommended that 
counselors be available to instructors and tutors for 
consultation on these issues. 

— Irene Doo, Associate Professor 

Math 

Diane Kramer, Professor 
Retention & Student Services Counseling 
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When the Chemistry’s Right— Web It! 

The World Wide Web (WWW) is growing in use 
as an alternative forum for instruction. Each time dis- 
tance education takes on a new technology, the purpose 
is usually to increase flexibility and/or accessibility of 
instruction. But educators must beware of whether such 
"tech gizmos" actually promote and sustain learning. 
Thus, evaluating students’ learning and attitudes is 
essential to assessing the effectiveness of this new tool. 

Due to concerns about whether students could learn 
such abstract concepts as those presented in a chemistry 
course via WWW, but still interested in web courses, I 
chose to write a small module of material for the web 
instead of a full semester's course. In doing so, I was 
able to do a portion of the work in preparation, but 
perform full-scale assessments for its viability. 

In order to make an online or web course, an in- 
structor must realize several important issues: 

• more time is required of the instructor for 
preparing the online course, as compared to 
face-to-face (in-class) instruction 

• more time is required of the instructor during 
the online course 

• to be well organized, the online course requires 
a great deal of planning 

• the instructor needs to be more aware of stu- 
dent concerns and attitudes while writing and 
delivering the online course (i.e. watch for 
misconceptions, don't leave student alienated or 
isolated, etc.) 

• the instructor does not have to work alone — 
technical personnel and instructional designers 
can offer much assistance 

Luckily, there are several sources of advice and 
pointers on how to design and implement an online 
course (see references). 

Methodology 

A database software program called WebCT 
was the backbone structure of the web course module. 
The WebCT software has several functions including 
homepage(s), chat, bulletin board, content pages, calen- 
dar, private email, keyword searching, online quiz soft- 
ware, and student tracking. These functions allowed 
me, as instructor, to build a very versatile and interac- 



tive web course and to track student progress through 
that course (and a knowledge of HTML was not a 
prerequisite). 

The design and development stage of the web 
course was the most time-consuming. Considerable ef- 
fort was made to instill in the course some "human- 
ness" for the students (and for myself). I wanted to make 
a course that would be more accessible than traditional 
courses, but that wouldn't leave the student feeling 
isolated or confused. Thus, several technologies were 
incorporated: 

• use of college-approved textbook 

• on-line content notes/instruction "lectures" via 
WWW (WebCT) 

• disk-based tutorials (see references) 

• asynchronous student to student and student to 
instructor discussions (bulletin board) 

• opportunity to send/receive documents between 
students and between students and instructor (email) 

• hardcopy handouts and worksheets 

The bulletin board in WebCT is like email, but 
where email is sent specifically to one person, the letters 
in the bulletin board are posted publicly, for all class 
members to see on the web. Also, where email is listed 
chronologically, the bulletin board is organized by topic 
and sub-topic. The bulletin board was utilized for both 
content and casual discussions. This mixture helped to 
liven the web site and give a personal feel to it- 
something that was appreciated by the students and 
myself 

I also wanted to make sure that this much time and 
effort on my part was "worth it". Therefore, I conducted 
surveys and evaluations of the students' attitudes and 
learning. That is the focus of this paper. 

The web-course module, which covered the con- 
cepts of atomic structure and the quantum model of the 
atom, was offered as an alternative for my students in 
Introduction to Chemistry, CHM 1614. For three weeks, 
five volunteer students participated in the web course 
instead of attending classroom instruction; however, 

(Continued on page 9) 
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they still attended weekly laboratories. Those five are the 
participants in the surveys which will be discussed next. 

A multi-question survey was conducted using online 
quizzes (WebCT) both before the student began the three- 
week module and after each completed the module. The 
two surveys differed slightly in composition, but consisted 
of multiple choice and free-response questions to assess 
student attitudes. (Surveys available upon request.) 

Results 

According to the pre-module survey, the students 
who volunteered for the web-based module expected to 
learn more than they would have in face-to-face instruction 
and expected to spend more time on the web course than if 
they had attended traditional instruction. These expecta- 
tions may be due, at least in part, to four of the five students 
having had prior experience with some form of distance 
education. 

The primary motivation for volunteering for the web 
course was "to try it out" or to overcome barriers of 
location and time. In addition, the students expressed their 
views of the primary benefits of online education: time 
flexibility, self-pacing, location, money saved by not com- 
muting, and ability to print out or re-read course materials. 
The primary drawbacks mentioned included: fear of isola- 
tion, fear of miscommunication, conception that the web- 
course would be impersonal, absence of gestures and the 
human element, and the "hassle of having to write every- 
thing out". When asked what their primary concern was in 
entering the online course, responses fit into three cate- 
gories: grades and assignments, expectations, and environ- 
mental distractions. 

Knowing the students' fears and expectations at the 
onset helped me to alter the course slightly to encourage 
more interaction among the students and make a conscious 
effort to promote open, clear communication of such issues 
as expectations (e.g. assignments) and grading criteria. 
Without such feedback from the students, I may not have 
succeeded as well in my communications with them. For 
example, I may not have realized their fears, which could 
have formed a barrier to their learning. 

Results Post-Course Survey 

After completing the three-week online module, 
students’ attitudes remained about the same. They still 



believed that ACC should offer more online courses and 
would recommend online courses to a friend. In addi- 
tion, they would recommend this online course to a 
friend. 

The students felt they had spent more time with the 
online course than they would have spent with tradi- 
tional instruction, but felt their time was "worth it". 
They also felt they learned more (one of five) or much 
more (four of five) than they would have in face-to-face 
instruction. 

After completing the online course, the primary 
benefits listed by the students included time flexibility, 
money saved, self-pacing, and ability to re-review con- 
tents, which are similar to the responses to the pre- 
course module questionnaire. Differences were the addi- 
tional benefits of "interacted with other students", 
"responsibility of learning rests more heavily on the 
student", "if you learn something, you truly learn it", 
and the supplemental materials (including handouts and 
informative web sites). The drawbacks included: "not 
being able to go online with everybody [and] chat", 
"lack of interaction with other students", "at own pace", 
"deadlines could have been clearer", "having to use 
written communication", and "it was easy to procrasti- 
nate". 

As for what the students felt to be the greatest 
motivation in completing the web course: grades. 

Comments 

The students appreciated the bulletin board 
(asynchronous discussion) and each read every posting. 
We discussed everything from movies to photons and, 
when misconceptions did arise, both the students and I 
responded to gently correct it. In fact, the students were 
adept at helping each other learn both the content and 
the technologies. And, while they felt they had adequate 
contact with me (the instructor), they yearned for more 
contact with their peers. This could be handled with a 
chat program. (WebCT's chat function was not an option 
during this study.) This point is mentioned to impress 
upon other instructors that students do in fact need and 
want both synchronous and asynchronous forms of com- 
munication with web courses. In fact, motivation is 
closely tied to interaction with regard to web-based 
instruction, according to the literature. 

(Continued on page 10) 
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The students who participated in the web course 
showed an increase in scores on a quiz given before and 
after the module, thus showing that learning occurred. It 
must be noted, however, that this was an extraordinary 
group of students. They consistently scored better on all 
exams in class (by averages), 4 out of 5 were science 
majors, and 4 out of 5 had prior experience with distance 
instruction. Thus, educators should realize that their 
student groups may vary in performance due to students’ 
prior experience and motivations. 

Conclusions 

With regard to the characteristics of the student 
participants in my web-based course, the web is a viable 
alternative form of instruction for abstract courses such 
as the Introduction to Chemistry course. Evaluation of 
the students in three stages (the onset of the web module, 
throughout the module via student emails, and at the 
completion of it) proved very helpful in providing feed- 
back about how the students felt (thus leading to infor- 
mation about student motivations), how the students 
learned, and what "worked" and what didn't. 
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The most consistent feedback the Office of 
Institutional Effectiveness received in response to 
its Evaluation questionnaire was, “Ife don Y know 
what services and products you provide'' 

Our office is really not a “secret agency” — 
Really! To help dispel that belief, on the last page 
of this publication you will find a list of the 
services we provide faculty and staff and the 
products (publications and reports) we produce, 
as well as the procedures for accessing them. 

Additionally, all publications from this office 
will carry our newly created “logo” that is printed 
below. This logo will help you identify OIE 
publications in a flash, so you’ll know where the 
publication came from and whom to con-^ 
tact regarding it. 






Program Review Schedule 1999-2000 

Applied Technologies, Multimedia and Public Service 

Photography Commercial Art Printing Commercial Music 

Arts and Humanities 

Developmental Reading Developmental Writing 
English as a Second Language 

Business Studies 

Office Administration Management 
Business Management Financial Management 

Computer Studies and Advanced Technology 

Quality Assurance Electronic Technology 
Computer Information Systems/Computer Science 

Health Sciences 

Diagnostic Medical Imaging-Sonography 
Diagnostic Medical Imaging— Radiology Medical Lab Technology 

Math and Sciences 

Mathematics (including Developmental Math) 

Chemistry 

Social and Behavioral Sciences 

History 
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Quest for Quality: Instructional Program Review 



OVERVIEW 

The Instructional Program Review Process fo- 
cuses on aspects of need, cost and effectiveness of 
instructional programs. Specifically, is the instructional 
quality of the program high enough to meet institution- 
ally set standards? 

What is a program? 

For purposes of program review, a program is an 
organizational unit within the college that provides 
instructional or support services. The process outlined 
here focuses on instructional programs. 

Purpose of Program Review Process 

• Continuously improve the program. 

• Assess strengths and weaknesses of a program as 
well as the opportunities and threats it faces. 

• Determine the capabilities of the program. 

• Break down barriers between areas of operation. 

• Ensure best uses of resources — fiscal, human, fa- 
cilities, equipment, technology. 

• Articulate the program’s mission and vision. 

• Provide the foundation for application to the 
Greater Austin Quality Council (Baldrige Crite- 
ria). 

• Respond to Board policy. 

Characteristics of Program Review Process 

The Instructional Program Review Process: 

• Provides for identification of improvements. 

• Includes standards of academic quality. 

• Allows for better decision-making, including mod- 
ifications, budget, strategic planning. 

• Is integrated into the Institutional Effectiveness 
process. 

• Is integrated into other external review process, 
including the THECB site review, SACS accredi- 
tation, and program-specific reviews required by 
other agencies. 

• Ensures that customer focus is integrated into the 
program. 

• Creates an opportunity for broad-based input from 
constituencies including students, staff, faculty, 
and employers. 

• Is a cyclical process with the expectation that there 
will be continuous review within the program. 

• Assesses the degree to which the program is fulfill- 
ing its mission and accomplishing its goals. 

EFFECTIVENESS REVIEW FALL 1999 



Administration of the Program Review Process 

Oversight of the Program Review Process in in- 
structional programs is the responsibility of the Instruc- 
tional Program Review Committee. Membership in- 
cludes one faculty member from each dean area. Ap- 
pointments are made biennially and members serve for 2 
years. The Associate Vice President for Institutional 
Effectiveness serves as an ad hoc member. This Com- 
mittee is chaired by a member of the Committee selected 
by the Executive Vice President for Instruction. 

The responsibilities of the Instructional Program 
Review Committee include: 

• Overseeing the Program Review Process to ensure 
that it occurs within the stated timeline. 

• Monitoring of the implementation of the Program 
Review Process. 

• Providing assistance to units involved in the pro- 
cess. 

• Evaluating the overall process. 

• Developing modifications as necessary to ensure the 
process functions effectively. 

• Developing modifications of the Indicators of Effec- 
tiveness, including related criteria, as needed. 

• Reviewing self-study reports. 

Guidelines for Implementation 

In consultation with the administration. Deans de- 
termine the order in which programs in their areas will 
be reviewed. New programs will be incorporated into 
the timeline as they are developed. In most cases, 
program review will not occur during the first 3 years of 
program implementation. 

Self Study Team 

In consultation with the Program Coordinator or 
Taskforce Chair, each Dean will designate a Self-study 
Team for each instructional area to be evaluated. The 
primary responsibilities of the Self-study Team are to 
collect and analyze information, develop the Self-study 
Report, and make recommendations. 

The Chair is the key member of the Self-study 
Team. The Chair’s responsibility begins during the 
Spring Term prior to the Self-study and continues 
through the next academic year until the final Self-study 
report has been submitted. 

(Continued on page 12) 
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Indicators of Effectiveness 

The basis of the Program Review Process is the 
analysis of information relating to a set of Core Indica- 
tors which have been identified by the Program Review 
Work Group and reflect input from a variety of 
sources. They include indicators that have been identi- 
fied by federal, state and regional agencies (including 
SACS, the LBB, and the THECB) as well as “best 
practices” identified in similar processes at other com- 
munity colleges. 

As part of the Self-study process. Self-study Teams 
in consultation with the Dean and Instructional Pro- 
gram Review Committee will select the specific indica- 
tors on which their analysis will focus. The only 
exception to this is a set of required indicators that are 
defined by SACS, the LBB, the THECB, other agen- 
cies, and/or institutional priorities. 

INSTRUCTIONAL PROGRAM REVIEW 
PROCESS 

The Self-study begins with an analysis of the infor- 
mation related to the core indicators selected by the 
Self-study Team. Then the Self-study Team considers 
the following: 

• Review of Mission/Vision. The Mission of the 
instructional area will be reviewed to ensure that it 
continues to reflect the need for the program/discipline. 

• Identification of Strength, Weaknesses, Op- 
portunities and Threats. As part of the Self-study, 
the program will conduct a SWOT to determine the 
strategic priorities that need to be addressed. 

• Integration of Baldrige Criteria (optional). 

Programs interested in using quality-based criteria or 
pursing one of the quality awards may do so as part of 
the Program Review process. Specific information is 
available through the Office of Institutional Effective- 
ness. 

• Recommendations. The Self-study Team 
should identify the major issues that need to be ad- 
dressed as a result of information gathered during the 
self-study process. It should then identify what will be 



done to address the issues identified. In most cases, 5 to 
10 recommendations will be made. 

• Action Plans. As part of the Self-study, the 
Taskforce Chair/Program Coordinator must develop Ac- 
tion Plans for each Recommendation. Action Plans are 
proposals for change and may serve as guidelines for 
future planning activities. 

• Institutional Effectiveness Measures. At least 
three Institutional Effectiveness Measures need to be 
identified. Assistance on developing these can be pro- 
vided by the Office of Institutional Effectiveness. 

• Implications for the Budget Process. The Task- 
force Chair/Program Coordinator should develop an 
outline of operational planning priorities for next budget 
cycle. This should be shared with the Dean and Associ- 
ate Vice President during the budget process. 

DOCUMENTATION NOTEBOOKS 

The Office of Institutional Effectiveness (OIE) will 
provide documentation notebooks for each instructional 
area going through the Instructional Program Review 
Process. These notebooks include data and materials 
from federal and state reports as well as those developed 
by the OIE. 

INSTRUCTIONAL PROGRAM REVIEW 
COMMITTEE MEMBERS 
ACADEMIC YEARS 1999 and 2000 

Lynn Beaman (Chair), 

Applied Technologies, Multi Media and Public Service 

Guadalupe Lopez-Cox, Arts and Humanities 

Dwayne Crowley, Business Studies 

Ken Peterson, Computer Studies / Advanced Technology 

Jere Hammer, Health Sciences 

Mary Parker, Math and Sciences 

Gaye Lynn Scott, Social and Behavioral Sciences 

— Martha Oburn 
Associate Vice President 
Institutional Effectiveness 
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Survey Savvy: If You Want to Know, Ask... Effectively! 



• Variety and convenience were the two things students 
taking Vocational Nursing via distance learning liked 
most about the course. 

• ACC Graduates rated their preparation for employ- 
ment or further education "Excellent” 

• Pharmacists in the Austin area prefer to hire a phar- 
macy technician who has completed a formalized, 
structured pharmacy technician training program 
over someone who has not. 

Each of these statements is based on data gathered by 
a survey. Surveys provide information that is useful in 
outcomes studies, information from which faculty may 
formulate course improvement strategies. 

However, surveys are not “magic.” Good survey 
design requires some thought and effort. When planning a 
survey, remember the following guidelines. If you will 
need their assistance, contact the OIE as early in your 
planning process as feasible; OIE staff can work with you 
to complete the survey tasks. 

Determine when you will need to report the data 
that are to be collected. The complete survey process 
usually takes about four to six months from first step to 
completing a report of the results. Creating a timeline 
from the finish date to the first task will ensure you 
complete the survey report on time. 

Identify the questions you want the data to answer. 

What is it you want to know from conducting the survey? 
These are the overall reasons for conducting the survey. 

Identify the data you need to answer those ques- 
tions. Do you need performance data, opinions, compar- 
isons based on some criteria, etc.? The data you need will 
form the foundation of the data collection instrument. 



Identify the people from whom you will gather 
the data. Responses from which group of people will 
provide you the most useful data? Surveying students 
to determine their satisfaction with job placement ser- 
vices, for example, is useful only if the students you 
survey have used the services of the job placement 
office. Also, if your questions require data from people 
who meet specific characteristics, this is the step 
wherein you identify those characteristics (students 
with 0-9 hours, students with 10-15 hours, students 
majoring in X, etc.). 

Determine when to conduct the survey. In gen- 
eral, the information you are collecting determines 
when you conduct the survey. If your major interest has 
to do with information about expectations of a course/ 
experience, for example, the survey will be conducted 
at the beginning of the semester/course/experience. If, 
on the other hand, you are seeking information regard- 
ing student experience of a course or with the use of 
services, the survey would provide most useful results 
at the end of the semester or after the use of the service. 
If you are measuring impact or change resulting from an 
experience you may wish to survey the selected group 
both before and after the experience. 

Design the methodology for conducting the sur- 
vey. This is the step where you decide the procedures 
for conducting the survey: how many people you will 
survey, how you will survey them (by phone, in class, a 
mailed questionnaire, etc.), how you will distribute and 
collect questionnaires, make follow-up contacts, pre- 
pare the questionnaires for data entry, etc. OIE can 
transform your questionnaire into a scan (bubble sheet) 
form for easier data analysis. 

(Continued on page 14) 



**A survey is a system for collecting information to describe, compare, or explain knowledge, attitudes, and be- 
havior. Surveys involve setting objectives for information collection, designing research, preparing a reliable and 
valid data collection instrument, administering and scoring the instrument, analyzing the data, and reporting the 

results. ” Arlene Fink 

The Survey Handbook. Thousand Oaks, Sage Publications, 1995. 
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(Continued from page 13) 

Design and produce the questionnaire. Creating a 
useful questionnaire, a topic about which many books 
have been written, requires careful thought and skillful 
application of some basic rules. Keep in mind a survey 
questionnaire should be as brief as possible (aim for no 
more than one side of a single page at most) and should 
create as little frustration as possible to increase the 
likelihood that it will be completed and returned. The 
aim of a useful questionnaire is to help the people you are 
surveying give you the information you need in a form 
that is useful. 

Include only those questions which are important to 
the current study. Don't ask for "age" if it is not pertinent 
to answering your outcomes question. 

Make the questions specific', avoid vague qualifiers 
and abstract terms. Terms like “usually,” “most,” and 
“now” are full of ambiguity; rather use “each day/week/ 
semester,” “4 of 5 times,” or “since you completed the 
class.” 

Start with easier questions and move to more difficult 
or boring ones. The first questions should be chosen 
with care. They should "hook" the reader into answering 
the survey questions. 

Ask questions in logical order. Avoid "contingency" 
questions; those where if you check "yes" to one ques- 
tion, you then "GO TO" another set of questions else- 
where. They are confusing and tend to lower the number 
of completed questionnaires returned. 

Construct response categories carefully. Response 
categories must allow for all possible responses yet not 
be too long. If you are asking students how much time 
they spend studying, you would want to include “never” 
as well as “X hours every day” but you would not want to 
list all the number of hours in a day. You would provide 
categories of hours within the day, such as “1-3 hours per 
day,” “4-6 hours per day,” etc. 

Provide clear and sufficient directions, including the 
reason for the survey, whether responses are to be anony- 
mous or confidential, how the respondent is to complete 
the questionnaire and what to do with it when they have. 
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In other words, as you are constructing the question- 
naire, keep in mind the last survey questionnaire you 
were asked to complete and design yours the way you 
wish it had been designed, i.e., from the perspective of 
those who will be completing it. 

When possible, have about ten people who are simi- 
lar to those you plan to survey complete the question- 
naire and give you feedback, then make improvements 
accordingly. Are the directions clear? Are the questions 
easy to understand? Does the format invite responses? 
How long did it take them to complete the questionnaire? 

Conduct the survey. Distribute questionnaires as 
outlined in the “methodology.” As they are returned, 
track the number completed. If your questionnaire was a 
scan form/bubble sheet, “clean up” the responses, i.e., 
make sure bubbles are completely filled in, use a pencil 
to go over any bubbles that were filled in with ink, erase 
any stray marks, and be sure all identifying data are 
complete (section number or course number for exam- 
ple). As the deadline for returning the questionnaire 
approaches, determine whether you will send a reminder 
to return the questionnaire. Reminders can be as simple 
as a postcard — Did Your Return the X Survey Question- 
naire Yet? — or more complex — another questionnaire 
and a reminder note or even a phone call to those people 
who have not returned the questionnaire requesting they 
do so “now” over the phone. 

Analyze the data. There are several software pack- 
ages available to analyze data, but if these are not avail- 
able to you, the OIE can help you with this task. 

Report the results. The final task of conducting a 
survey is to communicate the findings clearly and accu- 
rately so they can be used for making decisions. Your 
report should include a meaningful title. “Student Sur- 
vey Results” says little; “Factors Related to Student At- 
trition” says much more. To orient the reader to your 
report, include the purpose of the study and how the sur- 
vey was conducted (the methodology used). Provide a 
summary of your results, including any tables or charts 
displaying data. And finally, draw your conclusions and 
make recommendations based on your findings. 

— Roslyn Wallace 
Institutional Outcomes Coordinator 
Institutional Effectiveness 
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Q: How do we decide which group to target for a 
survey? 

A: The answer to this question depends on who your 
audiences are and what you want to know. Do you 
want to know how students learned about your pro- 
gram and what they expect from it? Then, a survey of 
students just entering the major is appropriate. Are 
you interested in exploring students: perceptions of 
the major program and how well it met their expecta- 
tions? Surveying graduating seniors would give you 
this information. Are you interested in determining 
how well students in your program were prepared for 
employment? Surveying employers of graduates of 
your program would provide that information. In 
other words, the group you survey would be selected 
based on their ability to provide data to answer the 
question your survey is trying to answer. 

Q: How can we ensure a good response rate on our 
surveys? 

A: One of the most crucial factors affecting response 
rate is the length of the survey. A survey should take 
only 15 minutes or so to complete. Appearance is 
important; the questionnaire should be easy to fill out, 
with clear and concise instructions. A cover letter that 
explains the purpose of the survey and assures confi- 
dentiality of the respondents’ answers can make a 
significant difference in whether the questionnaire is 
completed and returned, as well. If you are distribut- 
ing the questionnaire to be completed and returned at 
a later date, a postcard (or e-mail or phone call) 
reminder can boost response rate quite a bit. 

Q: Should our survey be anonymous? 

A: Surveys should be confidential: results are pro- 
cessed and reported in such a way that individual 
responses cannot be identified. However, some peo- 
ple will not respond to the questionnaire without the 



assurance of total anonymity. Four things should be 
considered when deciding whether to conduct the 
survey anonymously: 

• the nature of your questions — do they ask for 
information that respondents may be hesitant to 
share? 

• the people you are surveying — are they sympa- 
thetic to the goals of your survey or suspicious 
of your motives? 

• the amount of factual information you need — if 
the survey is anonymous, you will not be able to 
match additional information available from 
other data sources which you may wish to use in 
your analysis, e.g., GPA, credit hours earned to 
date, etc. The more you can get information of 
this type elsewhere, the shorter the questionnaire 
will be and likely, the greater the response rate. 

• the importance of follow-up mailings — a follow- 
up reminder for a “complete elsewhere” survey 
will be impossible if the survey is anonymous, 
however, if students are to complete the ques- 
tionnaire in class, there is no need for a follow- 
up reminder. 

Q: What survey tasks can be delegated to someone 
else? 

A: Duplication of the questionnaire and cover letter, 
assembly of the questionnaire “packets,” and track- 
ing returned questionnaires may be designated to 
someone else, but avoid “slicing up” the work of 
conducting a survey among too many people. Con- 
ducting a survey is one task where the saying, “Too 
many cooks spoil the broth” is very true. Invite 
input from others, but identify one, or at most two, 
people to be responsible for the overall conduct of 
the survey. A “task” list on which you check off 
each survey task as it is completed is a very useful 
tool for tracking the progress of the survey. And, of 
course, the Office of Institutional Effectiveness is 
always available to assist with your survey tasks. 
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OIE At Your Service 

To assist you in making decisions and assessing the effec- 
tiveness of your area, the Office of Institutional Effectiveness 
provides services to faculty, staff and administrators through 
reports and other publications produced on a regular basis, 
specialized assistance provided on request, and maintenance of 
a comprehensive web site that includes both the college-wide 
strategic planning database and the institutional effectiveness 
database as well as other resources for program review and 
studies of college-wide issues. 

Publications and Reports Provided on a Regular Basis 

• The Daily Registration Update is generated daily and 
posted to the OIE web site in .pdf format. This report 
displays current headcount, contact hours, and credit hours 
by college-wide totals as well as by campus location and 
Dean area. 

• The Ejfectiveness Update summarizes effectiveness mea- 
sures, criteria, and outcomes for college-wide missions 
such as transfer, workforce, student access, and student 
retention. This report is posted to the OIE web site and 
paper copies are distributed to all college employees. 

• The ACC Effectiveness Review is published once each fall 
and spring semester. This publication contains summaries 
of effectiveness studies conducted by ACC faculty and 
staff, assessment information, and a "Q&A" section. It is 
distributed to all college employees and also posted to the 
OIE web site. 

• The Enrollment Comparison is posted to the OIE web site 
each semester and compares headcount, demographic, 
credit hour, contact hour, and Full Time Student Equiva- 
lent (FTSE) data for all sessions of the current academic 
semester with those data for the same semester of the 
previous academic year. 

• The ACC Fact Book is published once each year. This 
document provides a comprehensive five-year snapshot of 
demographic information on students, faculty, and staff, as 
well as programs and facilities. It also includes budget and 
finance information and general information about the 
College— its history and mission, etc. Printed copies are 
distributed to Administrators, Task Force Chairs and Pro- 
gram Coordinators, the campus Provost's offices, and the 
LRS at each campus. It is also available on the OIE web 
site. Faculty and staff may request personal copies from 
the OIE. 



• The Preliminary Enrollment/High Demand/Head- 
count Attrition Reports are published each semester 
following the close of registration. This document 
provides information on student headcount, sections, 
enrollments, and contact hours, as well as the number 
of students who could not enroll in full sections (high 
demand courses) by discipline. 

Assistance Provided on Request 
The staff of the OIE can provide the following types of 
assistance to faculty and staff to assess the effectiveness of 
units or areas. 

• Defining Institutional Effectiveness measures and 
outcomes 

• Designing study methodology 

• Clarifying study questions. 

• Defining demographics needed for analysis. 

• Selecting population samples. 

• Determining study tasks and timelines. 

• Designing data collection instruments including eval- 
uation forms and survey questionnaires 

• Converting data collection instruments into scanable 
forms. 

• Analyzing and summarizing data you have collected, 
including student's course evaluations and feedback. 

• Conducting Focus Groups 

College-wide Services include assistance with program 
review, strategic planning, and the institutional effective- 
ness process. OIE coordinates federal and state reporting 
and can provide data for grant applications. 

The OIE Web Site 

OIE maintains a web site on ACCNet. All our 
publications, reports, and college-wide studies are posted 
to this site. As studies of college-wide issues are conducted, 
they are posted to the site as well. Other services provided 
on the OIE web site include maintenance of two databases: 
one for documentation of strategic planning activities and 
the other for documentation of the institutional effective- 
ness process. Additionally, you may use the on-line data 
request form or email us at oieinfo@austin.cc.tx.us to ask 
for customized assistance or information. The address of 
the OIE web site is 

http://www2.austin.cc.tx.us/oiepub 

— Roslyn Wallace 
Institutional Outcomes Coordinator 
Institutional Effectiveness 



intranet data request form: 



web site address: 



e-mail address: 






http://accweb.austin.cc.tx.us/oie/datareq.htm 



http://www2.austin.cc.tx.us/oiepub 



oieinfo@austin.cc.tx.us 
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Assessment.. .Because Learning Matters Most 



“Assessment 

is 

a rich 

conversation 

about 

student learning, 
informed 
by 

data. ” 



T. J. Marchese 



I X ssessment of student 
learning begins in the classroom. This 
is the setting in which, as the Latin root 
of the word assessment suggests, the 
teacher and student can sit beside each 
other to evaluate learning in a 
“conversation” that informs improve- 
ment. At the program level, a similar 
“conversation” occurs when faculty 
collaborate to identify, and then design 
assessment plans to evaluate, the criti- 
cal learning outcomes students should 
be able to demonstrate as a result of 
completing the program. And finally, 
at the institutional level, the 
“conversation about student learning” 
occurs in the evaluation of the effec- 
tiveness of college-wide functions 
such as transfer and workforce pro- 
grams, retention, student services, and 
developmental and adult education. 



Inside 
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Assessment 
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Book Review 
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Program 

Assessment 
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Assessment 
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Assessment vs. 
Grades 
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Common to all of these conversa- 
tions is the belief that learning is what 
matters most to those of us in commu- 
nity college education. Because of this 
belief, we assess our effectiveness; we 
seek evidence of the degree to which 
learning is taking place and informa- 
tion to guide the steps we must take to 
improve. 



The process by which we collect 
this evidence and information is our 
institutional effectiveness process: 

1. Identify the purpose (of the class, 
program, institution, etc.) 

2. Create Effectiveness Measures 

• the intended outcomes 

• the criteria 

• the methodology 

3. Conduct assessment and analyze re- 
sults. 

4. Implement Improvements. 

These steps are directly connected. 
Thus, the purpose is made concrete and 
specific through the identification of 
effectiveness measures for learning 
outcomes; the outcomes, in turn, are 
assessed; and improvements are imple- 
mented to enhance achievement of the 
purpose. Completing all these steps 
provides evidence of learning and in- 
formation to guide improvement of 
what matters most to us — student 
learning. 



— Roslyn Wallace 
Institutional Outcomes Coordinator 
Institutional Effectiveness 



Office of Instituti onal Effectiveness 

O 

ERIC 




27 





ACC Effectiveness Review 



In the following articles, several instructors share their experiences of ^'conversations about student learning'* 
informed by Classroom Assessment Techniques (CATs), from the book of the same name by Thomas A. Angelo and 
K. Patricia Cross. 



CATs Meow 

“Meow, meow,” said Patches 

“That’s all you do. Patches, is cry, cry, and 
cry,” I said. “What do you want? I fixed your 
food, opened the door, and even brushed you; and 
still you meow.” 

Does this sound like your class, are they never 
satisfied? At the start of each class there seem to be 
so many questions, all of which we covered the 
time before. Did I omit some material? Was the 
information clear, or should I have used a visual? 

I needed input from experts, the students. So I 
wrote my first CAT, Classroom Assessment Tech- 
nique, simple and to the point. 

7. What was unclear about the section we 
covered? 

2. What was the clearest point made about the 
same section? 

Those were the two questions given to my 
class. I told them they would be helping me to 
learn to teach better, which would help them to 
learn the material. Long faces looked back at me. 
There was a pause, “There will be no names, no 
grades, and I will review the comments and get 
back to you.” Long faces looked back again as I 
passed out the sheets with the two questions and 
left. Later I returned to pick-up the CAT. Wonder- 
ful, they all made comments; not all were the same, 
but with a little work they came together to form a 
picture of what was unclear and clear. 

During the next class, I reviewed the material 
they had submitted. Surprise, there were now 
happy faces smiling back at me and positive com- 
ments later. 



This experience was to play out twice more 
before the end of the semester. I, of course, 
changed the questions to fit the material covered 
on those days. The students were eager to fill me in 
on where I may have fallen short and thankful to 
have the missed information repeated. 

My experience was a real eye opener and idea 
creator. How the lecture was delivered changed, 
and use of visuals and demos became more com- 
mon. There was one unforeseen benefit: as the 
students became active and asked more questions, 
I became more at ease in front of the class. 

CATs paid off for me; they can for you also. 
Try one, it’s simple; I will even give you the 
questions. Give me a call, and see if the CAT’s 
Meow can tell you something. 

— David Tobey 
Assistant Professor 
Printing 

BIO CATs 



Teaching in ACC’s biology and chemistry and 
now the biotechnology program has led to my 
association with some outstanding teachers and 
with the faculty development office. Last year, I 
told John Hastings I wanted to learn new ways to 
assess my students’ learning during class, to get 
student feedback on whether I was doing a good 
job or not, and to help them discover the best ways 
for them to learn. He gave me the book Class- 
room Assessment Techniques by Angelo and 
Cross. This book contains numerous examples of 
classroom assessment techniques (CATs), and I 
would like to share with you several used by the 

( Continued on page 3) 
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(Continued from page 2) 

group who is developing the biotechnology pro- 
gram. 

In the book, CATs are divided into three cate- 
gories: techniques for assessing course-related 

knowledge and skills (content); techniques for as- 
sessing learner attitudes, values and self- 
awareness; and finally techniques for assessing 
learner reactions to instruction. We use techniques 
in all categories, but for this article, we decided to 
give one example for each category and share how 
we use it in our class. 

Alice Sessions is teaching BIO 1406 Cellular 
and Molecular Biology; she presents an example 
of category one. I am teaching BIT 1613 Introduc- 
tion to Biotechnology; I give an example of the 
second category. Finally, D’Maris Allen is teach- 
ing BIO 2420; she presents an example of the last 
category. 

1) Techniques for assessing course-related 
knowledge and skills: Content and Problem 

Solving - Isomers and Functional Groups 

Cellular and Molecular Biology (BIO 1406) is 
a rigorous course for science majors that is de- 
signed to articulate with the University of Texas, 
Southwest Texas State University and other four- 
year colleges. 

A strong working knowledge of chemistry is 
essential for success in this course. However, 
ACC teachers of Cellular and Molecular find that 
students have spotty backgrounds in chemistry. 
Some students took the course years ago, others at 
another institution. Sometimes, a fear of chem- 
istry itself prevents students from understanding 
the subject or speaking up in class when they do 
not know something. 

I resolved this difficulty by combining lecture 
with a variation on the CAT, “documented prob- 
lem solutions,” wherein student groups document 
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the solving of problems, grade their work, and get 
immediate feedback on what they have just 
learned. 

For example, I discuss isomers (structural, geo- 
metric and enantiomers) and functional groups as 
an introduction to macromolecules. After dis- 
cussing isomers and functional groups, I hand out 
four questions and ask students to document the 
answering of the problems, in student groups of 
three or four, using the text book and their notes. 
The papers are collected and graded, with each 
member of the group given the group’s grade. The 
questions emphasize higher level skills of analysis 
and synthesis and reinforce the skills just taught. 

The following are representative of the ques- 
tions asked of the groups: 

1. Draw the 4 structural isomers (skeletons) of 
C^Hi 2 (One is a ring; two are branched, one is 
unbranched) 

2. Draw and label the cis and trans isomers of 
2-butene, CH,CH=CHCH,. 

3. Serine is an amino acid. Look up serine in 
your text and draw it. Circle and identify the 
functional group(s). 

Synthesizing information from problems 1-3: 

4. Invent a molecule that contains 6 carbon 
atoms, a carbonyl group, a sulfydryl group and 
enough hydrogens so that every carbon has four 
bonds. It is your choice how the carbons are 
arranged and where the groups are attached. 

The presence of the group allows for peer 
teaching by those who are better prepared in chem- 
istry and relieves anxiety for all. When I used only 
handouts, overheads and other tools without im- 
mediate assessment, students frequently missed 
these questions on unit exams. However, when I 
added group work, documentation of the group's 
problem-solving technique, and immediate assess- 
ment, student comprehension and retention im- 
proved dramatically. 

(Continued on page 4) 
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(Continued from page 3) 

2) Techniques for assessing learner atti- 
tudes, values, and self-awareness: Produc- 
tive Study-Time Log - Studying for Tests 

I use the productive study-time log CAT to 
find out and help my students find out if they 
studied productively for a test. The productive 
study-time log has students keep a record of how 
much time they spend studying for a particular 
test, when they study, and how productively they 
study at various times of the day or night. 

The procedure for designing a log is to first 
determine what you want to track (e.g. time spent 
on task, how time was spent, times of the day, 
where they studied). Once you decide what to 
focus on, students create the simplest log sheet 
possible for recording the information. (I sug- 
gest you change the focus with each test, and that 
you try it out on yourself before you try it out on 
your students.) 

The day I assign the log, I have a couple of 
examples for students to examine in class, and we 
discuss what they should include and what they 
should not include. I do not let them take the 
examples home. Students bring their logs every 
lecture period before the test, and I initial them, 
even if there are only blank spaces. This helps to 
prevent students filling in their logs the day they 
are due. The last thing I have them do is circle 
what they consider to be their most productive 
period of that day. 

Logs are due the day of the test. I review 
them during the test, jotting down trends and 
figuring out total times. We review them as a 
group during the lecture period in which I return 
the tests to the class. Along with going over the 
test, we also go over how they studied for the test. 

Be careful of your feedback as you do not 
want to be negative if the student did not spend 
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the amount of time you felt they should (hard to 
believe, but not everyone shares your love of your 
subject or of learning). If you are negative, you 
probably will not get an honest response the next 
time you use this assessment technique. Try to 
report trends, etc. without bias, and let students 
draw their own conclusions. Students, because 
many of them are unaware of their study habits in 
terms of what works best for them, find the logs 
interesting. 



An example of a productive study-time log is 
displayed below: 



Date 


Study Period 
[time) 


Location 


What did you do? 

Circle what you consider was 
the most productive period . 


11/5 


2:30 am to 4:00 
am 


Home 


Read Textbook/wrote up 
practice test questions 


11/5 


2pm to 4pm 


Oil change 


Added more practice test 
questions 


11/5 


5:30pm-6:00pm 


Home 


Reviewed questions/ added 
some more questions 



3) Techniques for assessing learner reac- 
tions to instruction: Reactions to Revamping a 
Class - BIOL 2420 Microbiology 

CATs can also be used to determine students' 
feelings. Microbiology has been redesigned to 
include molecular techniques through inquiry 
based exercises in laboratory and create more di- 
versity in lecture assessments. Thus, there is a new 
grading system for both lab and lecture. By using 
a CAT, I was able to get feedback on how students 
like the new labs and grading systems. 

Similar questions were asked at different times 
to students in both lab and lecture sections. They 
were: 

1 . What do you think about the grading system 
for the course/lab? 

2. What do you like best about the lecture/lab? 

3. What would you like to change/do different? 

( Con tin ued on page 5 ) 
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Because part of the CAT is to share feedback 
with students, I type a report and make a trans- 
parency of it, then spend class time discussing the 
results. (This can be part of your teaching portfo- 
lio.) The results from students in the lecture class 
indicate students like the grading scheme, like 
using CDs and worksheets for learning during the 
lecture period, and had a variety of improvement 
suggestions. I try to implement some of these 
suggestions so students know I value their input, 
such as using more discussions in covering the 
lecture materials. 

Results from students in laboratory class 
showed more diversity of student feelings. The 
overwhelming majority liked the laboratory grad- 
ing scheme and the basic organization and proce- 
dures in lab. However, some fine-tuning will be 
done before the course is offered again, based on 
the LAB CAT. 

For example, I will try to re-sequence some of 
the activities as well as leave out several exercises 
in an effort to avert having students perform more 
than two laboratory exercises at the same time. I 
will also give more instruction on graphing, partic- 
ularly using semi-log paper. 

While CATs take some class time, it is time 
well spent in requesting and receiving feedback, 
which for the most part, is an honest reflection of 
students’ feelings and opinions due to the nature of 
the assessment procedures. If you are interested in 
learning what students think about some things you 
do in your classroom, try it! 

— D’Maris Allen, Professor, Biology 
Linnea Fletcher, Professor, Biology 
Alice Sessions, Project Coordinator, 

Biotechnology 



Letting the CATs Out of the Bag 



What probably frustrates teachers most is 
seeing their students unable to grasp the con- 
cepts, ideas, theories, formulas, knowledge be- 
ing taught. Even the most effective teacher 
knows that different classes respond differently 
to course content. So much depends upon the 
students’ backgrounds, biases, previous prepara- 
tion; indeed, at times it seems as though the 
teacher can do little to break down the learning 
barriers which surround some students. Simi- 
larly, at times it appears that students magically 
grasp ideas so quickly that the teacher wasn’t 
even a part of the process! Given these varied 
responses, how can a teacher gauge his or her 
own effectiveness to make certain that learning 
is actually taking place? 

Many learning theories address this question. 
However, theories aren’t really much help when 
it comes to a teacher facing daily a class which 
seems unable to learn course content. English 
teachers regularly face student writers who can- 
not, no matter how hard they try, develop writing 
ideas and maintain topic focus; math teachers 
work over and over with students who cannot 
master logical formulas; history teachers work 
with students who take copious notes during 
lecture and discussion but who can’t pass the 
tests. As teachers for whom learning matters 
most, how can we tell how well our students are 
learning in time to help them learn better? 

A recently developed hands-on method for 
assessing the effectiveness of teaching has made 
its way into the college classroom: the Class- 
room Assessment Technique (CAT). Thomas A. 
Angelo and K. Patricia Cross, the creators of 
CAT, summed up the general concept best in 
the first chapter of their book Classroom Assess- 
ment Techniques: A Handbook for College 

Teachers, (See Book Review, p. 7) “Classroom 
Assessment helps individual college teachers ob- 

(Continued on page 6) 
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(Continued from page 5) 

tain useful feedback on what, how much, and how 
well their students are learning.” 

What’s interesting about CAT is that it is 
based upon what many teachers already do in an 
informal way. That is, most faculty ascertain 
whether or not students are actually learning what 
they need to know. Certainly, one way to judge 
that is through tests. But gauging effective teach- 
ing and learning should be done before a test is 
given. For example, if a large majority of students 
in class fail an exam, the teacher usually steps 
back and evaluates both the exam and the in-class 
instruction to try and figure out why so many 
students failed. But, doing this after the exam is 
too late. Effective teachers try to assure student 
learning before the exam ever occurs. CAT does 
Just that. 

CAT has several important characteristics: it 
is learner-centered, teacher-directed, mutually 
beneficial, formative, context-specific, ongoing, 
and (most important of all ) rooted in good teach- 
ing practice. Thus it allows each faculty member 
to continuously (and painlessly) assess Just how 
much learning is occurring and why. 

Although the CAT system is an efficient and 
easy-to-use technique, trying to explain the details 
of it in this limited space would be futile. In fact, 
the bulk of Classroom Assessment Techniques 
consists of various examples of assessment tech- 
niques being used in different disciplines. The 
concept builds upon what most teachers already 
know: that simply asking students at the end of 
class if they have any questions probably won’t 
reveal whether or not learning is occurring. 

However, some of the easiest CAT techniques 
take only a few words to discuss. The first is The 
Minute Paper. Using this classroom assessment 
technique, the teacher asks the students at the end 
of the class period to respond to only two ques- 
tions: 1) What was the most important thing you 



learned today? 2) What questions remain upper- 
most in your mind as we conclude this class ses- 
sion? 

By reading the short responses, the teacher can 
check to see if she is getting the main points across 
and can identify what still puzzles her students. As 
a result, she can manipulate how she is teaching 
what. When she discovers what most students 
thought was important, she can ask herself two 
things: Is what students see as important what I see 
as important? If there’s a difference there, she can 
refocus her discussion during the next class period, 
making sure she emphasizes what she wants them 
to focus on. And she can ask herself why students 
still had questions about a topic she thought she 
clearly stressed. Do they need more fundamental 
review of the confusing point? Do they even have 
a sufficient foundation of knowledge to understand 
the fundamental point? 

Another quick technique is called The Muddi- 
est Point. With this approach, students are asked to 
briefly respond to this question: “What was the 
muddiest point in my lecture today?” Once the 
teacher knows what has remained muddy in her 
students’ minds, she can re-think her teaching ap- 
proach for that portion of material. Perhaps she’ll 
want to use visuals rather than a verbal explanation. 
Perhaps she’ll want to clarify the fuzzy point with 
group work activities. 

Other short techniques are The One-Sentence 
Summary, Directed Paraphrasing, and Applications 
Cards. In addition to these quick effectiveness 
gauges, CAT offers ideas for longer, more complex 
yet highly effective teaching and learning assess- 
ments, such as the Memory Matrix. 

The concept of teaching effectiveness is not 
new. Because faculty believe so deeply in the value 
of an education, they constantly work to make 
certain that their students are learning what they 
should and must know. As illustrated by the CAT 

(Continued on page 7} 
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system, measuring effectiveness does not have to 
be difficult. But, it will always be rewarding. 

If you’re interested in finding out more about, 
or getting help with, CATs, call the Faculty In- 
structional Development Office at 223-7667. We 
have copies of the CAT handbook to loan and there 
are workshops scheduled as well. Also check out 
the following web address: 

http://www.siue.edu/~deder/assess/catmain.html 

— Terry Stewart 
Director, Teaching Excellence 
Faculty Development Office 



Book Review: 

Classroom Assessment Techniques 

This book review is reprinted from the May, 1995 issue of 
the ACC Effectiveness Review with thanks to Betty Samford, 
NRSG who is no longer with the College. 

The handbook. Classroom Assessment Tech- 
niques, by Thomas A. Angelo and K. Patricia 
Cross, offers the college instructor a well written 
guide to data collection and evaluation to improve 
teaching effectiveness. The authors provide the 
reader with a blend of the philosophy, assump- 
tions, and goals of classroom assessment with 
concrete and fun ways to utilize the techniques in 
the didactic setting. 

The book is divided into four major parts: 

1) ways to get started; 

2) assessment techniques; 

3) research findings and future considerations; 

4) available resources. 

Part one includes an introduction to the Class- 
room Assessment Techniques (CATs), instruction 



in forming specific teaching goals using the 
Teaching Goal Inventory (TGI), and a three-step 
process of how to begin assessment in the class- 
room. Twelve examples, ranging from astron- 
omy to statistics, of previously used CATs are 
given to demonstrate how the techniques may be 
applied to a variety of disciplines. 

Part two details ways to select the right tech- 
nique and lists specific methods of assessing an 
assortment of cognitive and academic skills. For 
example, the “one sentence summary” technique 
can be used to assess a beginning nursing stu- 
dent’s ability to understand and synthesize the 
concepts of the nursing process. There are also 
examples of how to assess critical thinking, 
learner values, attitudes, and study skills. 

In part three, the authors discuss their findings 
regarding use of the TGI, CATs, and classroom 
research during a six-year period. They also 
address relevant ways to develop and approach 
classroom research using the goals and philoso- 
phy of the CATs. 

Part four contains a copy of the TGI for the 
reader’s self-assessment and a comparison of 
data between community colleges and four-year 
colleges using the TGI. 

This book is an invaluable resource for the 
instructor interested in improving the quality of 
classroom teaching. It provides creative and 
non-threatening methods of assessing teacher 
effectiveness. The CATs are structured to be 
used by one faculty member or in a team teach- 
ing approach in a variety of disciplines. 

— Betty Samford 
Nursing 

Classroom Assessment Techniques by Thomas A. An- 
gelo and K. Patricia Cross is available from Jossey-Bass, 
San Francisco. (ISBN 1-55543-500-3) 
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Program Level Conversations about Student Learning 



When you teach students in night classes, as we 
in the Legal Assistant Program do, the old saw that 
defines a professor as, “a person, who talks in your 
sleep,” comes a little too close to home. The eternal 
questions arise, “Are the students getting what they 
need?” “How much are they getting out of these 
courses, and how do we find out?” “How could we 
do it better?” These are of course, the ultimate 
questions asked by any educator, and though a good 
educator may have an instinctive feel for some of the 
answers, instinct does not readily translate into infor- 
mation accessible to all those who are asking the 
questions. The same questions are being asked by 
taxpayers, accrediting agencies, and, government 
entities charged with overseeing education. Ac- 
countability is the buzzword and, though we can rail 
against the current attempts to quantify everything of 
value, that won’t do us much good. Education is a 
main focus of the public’s cry for accountability. 

When the Legal Assistant Program was faced 
with the need to quantify how well it was doing its 
job of training paralegals, a number of approaches 
were considered; however, we realized early on that 
any evaluation would have to be based upon the 
goals and objectives of the Program. Put simply, the 
Legal Assistant Program’s primary goal is to do the 
best possible Job of preparing students to function as 
entry-level legal assistants/paralegals in a law firm 
or corporate/govemment law office. But how do we 
know if we’re succeeding? 

It took some time to realize that we didn’t have 
to wait until the students graduated and took Jobs in 
order to answer the question. One of the required 
courses in the Program is an internship. Since the 
internship is an opportunity for students to use in the 
“real world” what they learned in school, we don’t 
allow students to take the internship until they have 
completed the bulk of their coursework. The obvious 
solution to our problem was to let the attorneys who 
employ the interns, evaluate their preparation to 
function as entry-level legal assistants. We were 
already having the attorney supervisors fill out an 
individual evaluation of each intern that included 



multiple questions about the intern’s abilities. It was 
a very simple matter to add the following question: 
“Preparation to function as an entry level legal assis- 
tant” with a one to five ranking.. .one being 
“inadequate” and five being “superior.” We then 
asked those who should know, how well we were 
preparing our students to function as legal assistants, 
on a one to five scale. Of course, we also gave the 
evaluator an opportunity to comment on any of the 
evaluation criteria so as to let us know why a particu- 
lar part of the evaluation was very high or low. 

We now have a tool that tells us, every fall and 
spring semester, how well we are preparing our stu- 
dents; and where a student is not well prepared, we 
have individual questions that identify the deficiencies 
to help us improve our training. 

As an example, during a recent school year, over 
80% of the interns evaluated were deemed to have 
received “superior” preparation to function as entry 
level legal assistants and the balance were deemed to 
have “very good” preparation. Out of the nine areas 
where specific skills were evaluated, the average score 
was 4.41 out of 5. There was, however, concern 
expressed about the limited training in legal research 
on CD ROM and on line sources such as Westlaw. As 
a result of that concern, the Legal Assistant Program 
initiated training in CD ROM legal research and fol- 
lowed that up with comprehensive training in legal 
research using Westlaw. 

Obviously, an evaluation is only of use if it pro- 
vides information about what is being done correctly, 
and what could be improved. Our intern evaluation is 
not foolproof. When a student doesn’t put in the 
effort, the evaluation will reflect that lack of effort, but 
as educators, part of our Job description is to motivate. 
A particular evaluation may be telling us that we need 
to focus on that part of our Job. 

— Duane Crowley 
Professor and Program Coordinator 
Legal Assistant Program 

{see Legal Assistant Program assessment plan, opposite) 
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The Southern Association of Colleges and Schools (SACS) requires colleges it accredits to document their institu- 
tional effectiveness process. The ACC Unit-Level Institutional Effectiveness Database was designed to provide a 
centralized, web-based venue where all College units document their institutional effectiveness process. Each 
academic year, as units complete the steps in the institutional effectiveness process, information is entered into 
the database to provide a complete record of the unit's documentation, as required by SACS. Here is one assess- 
ment plan taken directly from the database. The Unit-Level Institutional Effectiveness Database resides on the 
ACCweb at http://salvador.austin.cc.tx.us/oiedb/intro.html 

Assessment Plan: Legal Assistant Program 

Academic Year 1997-98 Outcome #1: Job Preparation 

Intended Outcome: Students graduating from the ACC Legal Assistant Program will be prepared to 
function as entry-level legal assistants or paralegals. 

Assessment Criteria lA: Students taking the Internship Course during their last or next-to-last semester 
of the Program will be evaluated by a potential employer as adequately prepared to function as an 
entry-level legal assistant. 

Methodology: Employers of interns working in a law office or other legal environment are asked, 
at the end of the internship, to evaluate the intern’s performance in nine different areas and to determine 
whether the intern is adequately “prepared to function as an entry-level legal assistant.” The evaluation 
scale is 1 to 5, with 1 being “inadequate,” 2 being “needs improvement,” 3 being “adequate,” 4 being 
“very good,” and 5 being “superior.” 

Summary and Analysis of Data lA: During the 1997-98 school year, a total of thirteen students 
successfully completed an internship course. The students’ supervisors evaluated the students’ perfor- 
mance in nine different areas and also evaluated the students’ “preparation to function as an entry-level 
legal assistant.” In the area of “preparation to function as an entry-level legal assistant,” three interns 
were given “4s” and the rest were given “5s.” In the nine substantive areas, with 117 possible 
evaluations (13 students, 9 questions) there were 8 “N/As,” 4 “3s,” 40 “4s,” and 65 “5s” awarded. 

Improvement Actions lA: In the future we will add two questions to the evaluation in order to 
better determine what factors may have contributed to the four “3s” awarded to students. Since the “3s” 
were not concentrated in any single area, we need to see if other factors are affecting the result. We will 
ask for an evaluation of work ethic and attitude. 



Assessment Criteria IB: Less than 5% of the graduates of the Program will return to ACC to seek 
re-training in an area in which they completed a course with a grade of “C” or better. 

Methodology IB: Records of any graduates seeking retraining are kept. 

Summary and Analysis of Data IB: No students requested retraining. 

Improvement Actions IB: None 
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Institutional Level Conversations about Student Learning 

At the institutional level, the conversation about student learning takes place within the context of 
college-wide functions. Effectiveness measures for these functions provide evidence of the degree to which the 
outcomes promised in the institution's mission statement are being achieved. According to its mission statement, 
ACC provides vocational and technical programs leading to degrees, certificates, and jobs; freshman and 
sophomore courses in the arts and sciences; continuing adult education; compensatory education programs 
(developmental education courses and adult basic education); and assistance to students to help them achieve 
their educational goals. The Effectiveness Measures for these functions are presented here. Most were defined 
by the President's Effectiveness Council, composed primarily of faculty. However, some still in ''draft** status 
are currently under discussion in several College Councils. 

Each year, data defined in the methodology are collected, analyzed, and the results presented in the OIE 
publication. Effectiveness Update, as well as posted to the OIE web site (http://www2.austin.cc.tx.us/oiepub). 
Additionally, as the data are made available, the measures themselves may be revised to provide more useful 
information for these college-wide conversations about student learning. 



Workforce Education Effectiveness Measures 

Purpose: To prepare students for employment in industry and business. 



Intended Outcome 


Assessment Criteria 


Methodology 


1. Students in workforce education programs will meet their educational goals. 


lA. Associate 
degree seekers 


Analysis of student data will indicate that [of] workforce students 
who indicate their educational objective at entry is to "Complete 
an associate's degree, " 10% will complete a degree within 6 
years. Of those who do not complete a degree, 50% will achieve 
a certificate or marketable skills achievement award. 


Longitudinal analysis of student intent data 
and program/ course completion data. 


IB. Certificate seekers 


Analysis of student data will indicate that [of] workforce students 
who indicate their educational objective at entry is to "Complete 
a certificate," 10% will do so within 6 years. Of those who do 
not complete a certificate, 50% will achieve a degree or mar- 
ketable skills achievement award. 


Longitudinal analysis of student intent data 
and program/course completion data. 


1C. Non-graduates 


Analysis of student data will indicate that 90% of non-degree 
seeking students who are identified as Marketable Skills Achiev- 
ers and complete at least sbc semester credit hours in a workforce 
program will have a cumulative GPA of at least 2.00 when they 
leave ACC. 


Non-degree seeking students are those stu- 
dents who indicate on the initial application 
that (1) their educational objective at time of 
entry is "Take selected courses" and (2) their 
primary reason for attending ACC is "To 
improve skill for my present Job" or “To 
prepare for a future Job immediately after 
attending ACC." 


2. Graduates from workforce programs will find employment in jobs related to their fields of study. 


2A. Associate Degree 
recipients 


Within one year of graduation, 85% of AAS degree recipients will 
be employed in a Job that is directly or closely related to their 
field of study, or will be continuing their education. 


Annual surveys of graduates within one year 
of graduation and THECB data (as avail- 
able). 


2B. Certificate 
recipients 


Within one year of graduation, 85% of certificate recipients will 
be employed in a Job that indirectly or closely related to their 
field of study or will be continuing their education. 


Annual surveys of graduates within one year 
of graduation, and THECB data (as avail- 
able). 



President’s Effectiveness Council approved 1/12/98 




PAGE 



10 



36 



ACC EFFECTIVENESS REVIEW SPRING 2000 



ACC Effectiveness Review 



Academic Programs Transfer Effectiveness Measures 
Purpose: To prepare students for successful college or university Transfer 



Intended Outcome 


Assessment Criteria 


Methodology 


1. Transfer Rate 


lA. First-time in College (FTIC) students 
who have earned at least 15 credit hours in 
baccalaureate transfer courses will transfer 
to a four-year college or university within 
four years of their initial enrollment. 


ACCs transfer rate for FTIC students with a 
declared major in a transfer program will be 
higher than the state average transfer rate. 


Comparison of ACC and statewide FTIC 
transfer rates as published in the THECB 
Community College Transfer Rate Study . 


IB. Minority FTIC students who have 
earned at least 15 credit hours in baccalau- 
reate transfer courses will transfer to a 
four-year college or university within four 
years of their initial enrollment. 


ACCs transfer rate for minority FTIC stu- 
dents with a declared major in a transfer 
program will be higher than the state average 
transfer rate. 


Comparison of ACC and statewide FTIC 
transfer rates as published in the THECB 
Community College Transfer Rate Study. 


2. Transfer Intent Fulfillment 


2A. First- time students who indicate an 
intent to transfer and who have earned at 
least 15 credit hours in baccalaureate trans- 
fer courses will transfer to a four-year 
college or university within six years of 
their initial enrollment. 


65% of first-time students who indicate an 
intent to transfer and who have earned at 
least 15 credit hours in baccalaureate transfer 
courses will transfer to a four-year college or 
univenity within six years. 


Longitudinal analysis of student intent data 
collected from student applications. (First 
report Fall 2000) 


2B. Minority first- time students who indi- 
cate an intent to transfer and who have 
earned at least 15 credit hours in baccalau- 
reate transfer courses will transfer to a 
four-year college or university within six 
years of their initial enrollment. 


65% of minority first-time students who in- 
dicate an intent to transfer and who have 
earned at least 15 credit hours in baccalaure- 
ate transfer courses will transfer to a four- 
year college or university within six years. 


Longitudinal analysis of student intent data 
collected from student applications. (First 
report Fall 2000) 


3. Success at the Transfer Institution 


3A. ACC Students who transfer will be in 
good academic standing at the transfer in- 
stitution. 


70% or more of ACC transfer students will 
be in good academic standing at the transfer 
institution. 


Analysis of transfer data from transfer insti- 
tutions including UT at Austin, SWTSU, and 
TAMU. 


3B. ACC students who transfer will earn 
CPAs comparable to other transfer stu- 
dents at the transfer institution. 


The average GPA of ACC transfer students 
and other transfer students at principal trans- 
fer institutions will not be significantly dif- 
ferent. 


Analysis of transfer data from transfer insti- 
tutions including UT at Austin, SWTSU, and 
TAMU. 



President’s Effectiveness Council approved 10/5/98 
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Developmental Education Effectiveness Measures 

Purpose: To provide campus-wide programs and services that assist academically under-prepared students so that they will meet 
their educational goals. 



Intended Outcome 


Assessment Criteria 


Methodology 


1. Fall-to-Spring Retention Rates 


l.Al First-Time in College (FTIC) degree 
seeking students enrolled at ACC in the fall 
semester who are required to take develop- 
mental courses will return the following 
spring semester (Fall-to-Spring Retention) . 


Based on data provided by the THECB, the percentage of 
FTIC degree-seeking students enrolled in developmental ed- 
ucation courses in the fall semester who return the following 
spring semester will not be significantly different from the 
percentage of students who were not enrolled in develop- 
mental education courses who returned in the Spring. 


Using THECB Annual Data Pro- 
file report, compare the re- 
enrollment rates of students tak- 
ing developmental courses with 
the rate of those who are not. 


1.A2 First-Time in College (FTIC) degree 
seeking students enrolled at ACC in the fall 
semester who are required to take develop- 
mental courses will return the following 
spring semester (Fall-to-Spring Retention) . 


Based on data provided by the THECB. the percentage of 
FTIC degree-seeking students enrolled in developmental ed- 
ucation courses in the fall semester who return the following 
spring semester will not be significantly different from the 
state average for students enrolled in developmental educa- 
tion courses. 


Using THECB Annual Data Pro- 
file report, compare ACC and 
statewide average Fall-to-Spring 
retention rates. 


l.B There will be no significant differences 
by demographic group between Fall-to- 
Spring retention rates for FTIC degree or 
certificate seeking students enrolled in de- 
velopmental education courses and those 
NOT enrolled in developmental education 
counes. 


Based on data from the THECB Annual Data Profile, for each 
demographic group, the Fall-to-Spring return rate for FTIC 
students enrolled in developmental education courses will be 
within ±5% of the Fall-to-Spring return rate for students 
NOT enrolled in developmental education courses 


Using data from the THECB 
Annual Data ProPile report, 
compare developmental and 
non-developmental students’ 
average Fall-to-Spring retention 
rates by demographic group. 


2. Program Completion Rates 


2. A Students who are required to be en- 
rolled in Developmental Education Courses 
at ACC will complete state-mandated re- 
quirements.. 


Based on data provided by the THECB annually, the percent- 
age of ACC students enrolled in developmental education 
courses who complete state-mandated requirements will be 
at or above the completion rates for those students statewide. 


Using THECB data for LBB 
measures, compare ACC and 
statewide percentages. 


2.B Students who are required to enroll in 
developmental courses will complete their 
developmental requirements. 


Baseline data will be collected to determine the rates of 
completion for the following groups of students required to 
enroll in developmental courses: those required to take 

developmental courses in one area, those required to take 
developmental courses in two areas, and those required to 
take developmental courses in three areas. 


Using data from the student 
database, compare original pro- 
ficiency levels to last profi- 
ciency levels (below **5” indi- 
cates requires developmental 
courses; **5” or higher indicates 
completed developmental re- 
quirements) for each area by 
“group” (developmental courses 
required in one. two or three 
areas). 


2.C There will be no significant differences 
between developmental program comple- 
tion rates by demographic group. 


Based on data from the student database, the differences in 
developmental requirements completion rates among demo- 
graphic groups will be within ±5%. 


Using data from the student 
database, compare original pro- 
ficiency levels to last profi- 
ciency levels ((below “5” indi- 
cates requires developmental 
courses; “5” or higher indicates 
completed developmental re- 
quirements) for each area by 
“group” (developmental courses 
required in one, two or three 
areas) and by ethnicity. 
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(Continued from page 12) 



3. Course Completion Rates 


3. Students who are required to be 
placed in developmental courses 
will complete their courses at rates 
similar to those taking credit 
courses. 


Based on data from the student database, course 
completion rates (the percentage of students 
earning a grade of “C” or better) for develop- 
mental courses will be within ±5% of course 
completion rates for credit courses. 


Using internally generated reports, compare the aver- 
age course completion rate for developmental courses 
with that of credit courses. 


4. Developmental to Credit Mi| 


gration Rates 


4.A1 Students who successfully 
complete Developmental Writing 
will be successful in the next credit- 
level English course. 


Based on data from the student database, at least 
75% of students who complete Writing Skills II 
with a grade of "C" or better and then take 
English Composition I within two years, will 
complete English Composition I with a grade of 
"C" or better. 


Each semester identify cohorts of students who com- 
plete Writing Skills II with a grade of "C" or better, 
starting in Fall 1997. Each semester thereafter, track 
the cohort enrollment in English Composition I. Cal- 
culate the percentage of those who complete English 
Composition I with a grade of "C" or better within 
two years of their completion of Writing Skills II. 


4A.2 Students who successfully 
complete Developmental Writing 
will be successful in the next credit- 
level English course. 


Based on data from the student database, stu- 
dents who complete Writing Skills II with a 
grade of “C” or better and then take English 
Composition I or English Composition II within 
two years will have average course grades within 
±0.3 grade points of the average course grades 
for all students taking these courses who were 
not required to take developmental writing. 


Each semester identify cohorts of students who com- 
plete Writing Skills II with a grade of 'X" or better, 
starting in Fall 1997. Each semester thereafter, track 
the cohort enrollment in English Composition I and 
II. Compare the course grades for these students with 
the course grades for students taking English Compo- 
sition I and II who did not take Writing Skills II. 


4.B1 Students who successfully 
complete Developmental Reading 
will be successful in the next credit- 
level course. 


Based on data from the student database, at least 
75% of students who complete Reading Skills II 
with a grade of X” or better and then take a 
course designated to meet TASP guidelines 
within two years will complete the credit-level 
course with a grade of X” or better. 


Each semester identify cohorts of students who com- 
plete Reading Skills II with a grade of ’X" or better, 
starting in Fall 1997. Each semester thereafter, track 
the cohort enrollment in credit-level courses desig- 
nated to meet TASP guidelines. Calculate the per- 
centage of those who complete these courses with a 
grade of "C " or better within two years of their 
completion of Reading Skills II. 


4.B2 Students who successfully 
complete Developmental Reading 
will be successful in the next credit- 
level course. 


Based on data from the student database, average 
course grades of students who complete Reading 
Skills II with a grade of X” or better and then 
take a course designated to meet TASP guide- 
lines within two years will be within ±0.3 grade 
points of the average course grades for all stu- 
dents taking these courses. 


Each semester identify cohorts of students who com- 
plete Reading Skills II with a grade of "C" or better, 
starting in Fall 1997. Each semester thereafter, track 
the cohort enrollment in courses designated to meet 
TASP guidelines. Compare the course grades for 
these students with the course grades for students 
taking these courses who did not take Reading Skills 
II. 


4. Cl Students who successfully 
complete Developmental Mathe- 
matics will be successful in the next 
credit-level math course. 


Based on data from the student database, at least 
75% of students who complete Intermediate Al- 
gebra with a grade of “C” or better and then take 
College Algebra, Trigonometry, or Topics in 
Mathematics within two years will complete the 
credit-level course with a grade of “C” or better. 


Each semester identify cohorts of students who com- 
plete Intermediate Algebra with a grade of "C" or 
better, starting in Fall 1997. Each semester there- 
after, track the cohort enrollment in College Algebra 
or Trigonometry or Topics in Mathematics. Calcu- 
late the percentage of those who complete these 
courses with a grade of ’X" or better within two years 
of their completion of Intermediate Algebra. 


4.C2 Students who successfully 
complete Developmental Mathe- 
matics will be successful in the next 
credit-level math course. 


Based on data from the student database, average 
course grades of students who complete Inter- 
mediate Algebra with a grade of X” or better 
and then take College Algebra, Trigonometry, 
or Topics in Mathematics within two years will 
be within ±0.3 grade points of the average 
course grades for all students taking these 
courses. 


Each semester identify cohorts of students who com- 
plete Intermediate Algebra with a grade of "C" or 
better, starting in Fall 1997. Each semester there- 
after, track the cohort enrollment in College Algebra. 
Trigonometry, or Topics in Mathematics. Compare 
the course grades for these students with the course 
grades for students taking these courses who did not 
take Intermediate Algebra. 
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Adult Education Effectiveness Measures 

Purpose: To improve educational opportunities for adults 'svho lack the level of literacy skills requisite to effective citizenship 
and productive employment. 



Intended Outcome | Assessment Criteria Methodology 


1. Completion Rate: Adults who need basic education skills necessary for literacy functioning will complete the literacy 
program. 


1.1. Beginning Literacy- ABE students will 
complete beginning level adult basic educa- 
tion courses. 


15% of Beginning Literacy- ABE students will 
advance to intermediate level ABE courses. 


Using data reported to the state, calculate the 
percent of all Beginning Literacy- ABE students 
who advance to the intermediate level of adult 
basic education courses. 


1.2. Beginning Literacy-ESL students will 
complete beginning level adult basic educa- 
tion courses. 


17% of Beginning Literacy-ESL students will 
advance to intermediate level ABE courses. 


Using data reported to the state, calculate the 
percent of all Beginning Literacy-ESL students 
who advance to the intermediate level of adult 
basic education courses. 


1.3 Beginning ABE students will complete 
beginning level adult basic education 
courses. 


20% of Beginning ABE students will advance 
to intermediate level ABE courses. 


Using data reported to the state, calculate the 
percent of all Beginning ABE students who 
advance to the intermediate level of adult basic 
education courses. 


1.4 Beginning ESL students will complete 
beginning level adult basic education 
courses. 


22% of Beginning ESL students will advance to 
intermediate level ABE courses. 


Using data reported to the state, calculate the 
percent of all Beginning ESL students who 
advance to the intermediate level of adult basic 
education courses. 


2. Completion Rate: Adults who complete intermediate basic education will have sufficient basic education to enable 
them to benefit from job training and retraining programs and obtain productive employment. 


2.1 Intermediate ABE students will com- 
plete intermediate level adult basic educa- 
tion courses. 


23% of Intermediate ABE students will ad- 
vance to advanced level ABE courses. 


Using data reported to the state, calculate the 
percent of all Intermediate ABE students who 
advance to the advanced level of adult basic 
education courses. 


2.2 Intermediate ESL students will com- 
plete intermediate level adult basic educa- 
tion courses. 


23% of Intermediate ESL students will ad- 
vance to advanced level ABE courses. 


Using data reported to the state, calculate the 
percent of all Intermediate ESL students who 
advance to the advanced level of adult basic 
education courses. 


3. Completion Rate: Adults who desire to continue their education will advance to at least the level of completion of 
secondary school. 


3.1. Advanced ABE students will obtain a 
GED. 


30% of Advanced ABE students will obtain a 
GED. 


Using data reported to the state, calculate the 
percent of all Advanced ABE students who 
obtain a GED. 


3.2. Advanced ESL students will meet com- 
pletion standards set by ACC's college-wide 
ESL Task Force. 


24% of Advanced ESL students will meet 
completion standards set by ACC’s college- 
wide ESL Task Force. 


Using data reported to the state, calculate the 
percent of all Advanced ESL students who 
meet completion standards set by ACC’s 
college- wide ESL Task Force. 


4. Access 


4.1. Student enrollments in ABE classes will 
meet target numbers. 


Enrollments in ABE classes will be within 5% 
of the target set in the grant for each level of 
instruction. 


Using data reported to the state, compare 
actual enrollments in each level of instruction 
with target enrollments defined in the grant. 


4.2. Students enrolled in ABE classes will 
meet eligibility criteria. 


100% of students enrolled in ABE classes v^ll 
meet state eligibility criteria. 


Using data reported to the state, calculate 
percentage of students enrolled in ABE classes 
who meet state criteria. 


4.3. Students who take ABE classes will 
complete the initial 12 hours of class. 


80% of all students enrolled in ABE classes at 
ACC will complete the initial 12 hours of 
class. 


Using data reported to the state, calculate the 
percent of all students enrolled in ABE classes 
who complete the initial 12 hours of class. 
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PAGE 



14 



40 



ACC EFFECTIVENESS REVIEW SPRING 2000 



ACC Effectiveness Review 



Student Services Effectiveness Measures 

Purpose: To provide exemplary programs that support the teaching/leaming process and to increase opportunities for students to 
define and reach their educational goals. 



Intended Outcome 


Assessment Criteria 


Methodology 


1. Enrollment 


The undupUcated headcount enrollment of 
ACC will reflect the diversity of the college 
age population the College serves. 


The percent of ACC students from each demo- 
graphic group will be within ±5% of their propor- 
tion in the service area college age population. 


Compare THECB demographic data on 
the service area population and the fall 
undupiicated headcount enrollment of 
ACC students. 


2. Applicants 


The applicants who register and attend ACC 
will reflect the demographics of the appli- 
cant pool. 


At least 75% of each demographic group repre- 
sented in the applicant pool will actually enroll. 


Compare demographic data from the 
yearly applicant pool and the annual en- 
rollment of first time at ACC students. 


3. Financial Aid 


ACC will provide financial aid opportunities 
to students who demonstrate the most need. 


At least 50% of the students who receive Pell Grants 
will have an Expected Family Contribution (EFC) of 


Analysis of data from the Federal Pell 
Grant Program Student Payment Sum- 




zero (0) . 


maiy. 


4. Loan Default Rate 


The college will maintain a loan default rate 
5% below the maximum threshold required 
by law. 


The College's loan default rate, as calculated by the 
Default Management Division of the Department of 
Education, will be at least 5% less than the federally 
defined maximum threshold. 


Analysis of data from the Department of 
Education's Default Management Divi- 
sion's Annual Report io ACC. 



President’s Effectiveness Council approved 2/1/99 



Retention Effectiveness Measures 

Purpose: To provide consistent, campus-wide programs and services that increase the likelihood that students will remain in college 
and complete their degree or certificate program. 



Intended Outcome 


Assessment Criteria 


Methodology 


1. Fall-to-Spring Retention Rates 


lA. First-Time in College (FTIC) degree 
seeking students enrolled at ACC in the fall 
semester will return the following spring 
semester (Fall-to-Spring Retention) . 


The percentage of FTIC degree seeking stu- 
dents enrolled in the fall semester who return 
the following spring semester will be higher 
than the state average. 


Using THECB Annual Data ProGIe report, 
compare ACC and statewide average Fall-to- 
Spring retention rates. 


IB. There will be no significant differences 
between ACC and statewide average Fall-to- 
Spring retention rates for FTIC degree or 
certificate seeking students by demographic 
group. 


For each demographic group, the percent of 
FTIC students enrolled in the fall semester 
who return the following spring semester will 
be within ±5% of their proportion statewide. 


Using data from the THECB Annual Data 
Profile report, compare ACC and statewide 
average Fall-to-Spring retention rates by de- 
mographic group. 


2. FalFto-Fall Retention Rates 


2A. Students enrolled at ACC in the fall 
semester will return the following fall 
semester (Fall-to-Fall Retention) . 


The percentage of ACC students enrolled in 
the fall semester who return the following fall 
semester will be higher than the state average. 


Using THECB Student Migration report, 
compare ACC and statewide average Fall-to- 
Fall retention rates. 


2B. There will be no significant differences 
between ACC and statewide average Fall-to- 
Fall retention rates by demographic group. 


For each demographic group, the percent of 
ACC students enrolled in the fall semester 
who return the following fall semester will be 
within ±5% of their proportion statewide. 


Using data from the THECB Student Migra- 
tion report, compare ACC and statewide av- 
erage Fall-to-Fall retention rates by demo- 
graphic group. 


3. Completion Rates 


3A. FTIC degree or certificate-seeking stu- 
dents will complete their degree or certifi- 
cate within three years. 


Three-year completion rates of FTIC degree- 
seeking students will be at least 5% above the 
state average. 


Using data from the THECB Annual Data Pro- 
file report, compare ACC and statewide aver- 
age completion rates. 


3B. There will be no significant differences 
between ACC and statewide degree or cer- 
tificate completion rates of FTIC students by 
demographic group. 


For each demographic group, the percent of 
FTIC students who complete a degree or cer- 
tificate will be within ±5% of their proportion 
statewide. 


Using data from the THECB Annual Data Pro- 
file report, compare ACC and statewide com- 
pletion rates by demographic group. 
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Assessment vs. Grades 



Assessment 


Grades 


Formative 


Summative 


Diagnostic 


Final 


Non-Judgmental 


Evaluative 


Private 


Administrative 


Often Anonymous 


Identified 


Partial 


Integrative 


Specific 


Holistic 


Mainly Subtext 


Mostly Text 


Suggestive 


Rigorous 


Usually Goal-Directed 


Usually Content-Driven 



Annotations 

• Formative refers to the formation of a concept or item 
whereas summative refers to an "adding-up" or sum- 
mary stage. Assessments usually occur in mid-progress 
when corrections can be made. Grades are usually 
recorded at the end of a project or class in order to 
summarize academic quality. 

• Assessment is non-judgmental in the sense that it 
focuses on learning, which is the outcome of many 
influences, including teaching style, student motiva- 
tion, time on task, study intensity, and background 
knowledge. Therefore, no one element can be reason- 
ably singled out for praise or blame for a particular 
learning outcome. In contrast, grades carry evaluative 
weight as to the worthiness of student achievement and 
are applied, for good or ill, directly to them. 

• Assessments tend to be used in private and become 
public only under the assessor’s control. Grades, while 
not truly public, are part of the administrative record 
available throughout an educational institution. 

• Assessments are almost always collected in anony- 
mous fashion and the results are released in the aggre- 
gate. Grades are identified with specific students. 



• To use a metaphor from calculus, assessment more 
resembles a partial derivative whereas grades are 
more recognizable as an integrative process. 

• Assessment tends to look at specific parts of the 
learning environment. Grades are holistic in the sense 
that they record academic achievement for a whole 
project. Final grades, of course, can reduce academic 
achievement for an entire semester to a single mark. 

• The text of a course is its disciplinary content; grades 
tend to focus on that. The subtext of a course involves 
the transferable baccalaureate skills, such as critical 

• thinking, creative thinking, writing, and analysis. For 
example, the "text" of a course in anatomy and physi- 
ology includes the names bones and functions of 
muscles. The "subtext" of such a course might include 
scientific thinking, problem solving, and memory im- 
provement. Grades tend to focus on text; assessment 
tends to emphasize subtext. 

• Assessment findings tend to be suggestive and have 
pedagogical significance. That is, assessment findings 
shift pedagogy for reasons that need not be justified 
statistically, but can be justified when even one stu- 
dent learns better. In contrast, grades are recorded in a 
rigorous manner that does have statistical signifi- 
cance. 

• As with text and subtext mentioned above, grades tend 
to reflect student control of disciplinary course con- 
tent whereas assessment usually aims at the goals for 
all baccalaureate students, such as synthetic thinking 
and esthetic appreciation. 

— Doug Eder, Ph. D. 
Associate Professor, Neuroscience 
Director, Undergraduate Assessment & 

Program Review 
Southern Illinois University at Edwardsville 
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Employee Satisfaction with 

Services of Highly Used College-wide Offices 



''ACC 

offices 

need 



to 






ocus 



on 



X 



customer 



service. 



Anonymous respondent 



o provide the College with informa- 
tion to enhance its ability to meet the needs 
of its employees, an Employee Satisfaction 
Survey was administered in Spring 2000. 
The results of this survey were analyzed to 
determine which offices of the College were 
best meeting employee expectations and 
which were perceived as being less respon- 
sive. All offices were provided with a de- 
tailed analysis of their data and will develop 
plans to address the concerns expressed by 
ACC employees. This report will be limited 
to findings for “highly used” (80 or more re- 
spondents requested or received services) 
College-wide offices/services. 

Survey Development and 
Administration 

The Spring 2000 Employee Satisfaction 
Survey was based on a form that had been 
developed and administered in 1996. In Fall 
1999, it was modified to reflect the then cur- 
rent organization of the college and reviewed 
and approved by the Executive Vice Presi- 
dents and the Employee Association Presi- 
dents. (PLEASE NOTE: the College or- 
ganization has changed since the form was 
developed. However, all discussion and 
analyses are based on the organization of the 
College in Fall 1999 when the form was de- 
veloped.) 

The survey form was divided into two 
main sections covering College-wide ser- 



vices and Campus-specific services. The 
College-wide services included 63 offices or- 
ganized by administrative area; Campus- 
specific services included 29 campus-based 
offices. For each office, respondents were 
given an opportunity to indicate if they had 
requested or received services during the past 
year. They then rated the Promptness, Qual- 
ity, Attitude and Overall Service of the of- 
fices with which they were familiar. A space 
for comments was provided. 

In March 2000, the questionnaire was 
mailed to 2,903 ACC employees. It was 
mailed to home addresses to ensure that those 
without campus mailboxes would receive it. 
The employee address lists were provided by 
Human Resources and included all Full-time 
and Adjunct faculty, as well as all Profes- 
sional/Technical, Classified, and Administra- 
tive employees. In addition, the form was 
sent to a randomly selected sample of 306 
hourly employees and 392 Continuing Educa- 
tion and Adult Education Faculty. 

(Continued on page 2) 
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Results 

The overall return rate was 15.4% (447 returned 
questionnaires), but rates varied greatly by employee 
group. The highest return rate was 42.9% of the 28 ad- 
ministrators. About one-fourth of the Full-time Faculty 
and Professional/Technical staff returned the question- 
naire. One-fifth of the classified staff responded. The 
Hourly and Continuing Education/Adult Education fac- 
ulty had the lowest return rates, 2.9% and 1.3% respec- 
tively. The forms were scanned into a data file and data 
were aggregated by office for each question. 

Response rates for individual offices varied. Many 
offices did not have high numbers of respondents. The 



number of respondents requesting or receiving services 
ranged from a high of 280 for the Payroll office to a low 
of 10 for the Government Relations Office. The mid- 
point for all offices was 79 (17.7%) of the total 447 re- 
spondents. At least 80 respondents reported using ser- 
vices of 30 of the 63 College-wide offices listed on the 
questionnaire. For purposes of this report, these offices 
are referred to as “highly used.” 

A number of comments expressed concern about the 
survey instrument — that it was overly long and cumber- 
some. As one respondent commented, "This survey is foo 
busy' to complete in a 'few minutes'." Another indicated, 
“This survey was too long. Next time ask only about 10 ques- 

( Continued on page 3) 



Table 1; Ratings for Overall Service of Highh 


/-used College-wide Offices/Services 


College-wide Offices/Services 


Total=447 


dissatisfied 


neutral 


satisfied 


n 


% 


% 


n 


% 


n 


% 


n 


Payroll Office 


280 


62.6 


34.6 


97 


23.6 


66 


41.8 


117 


Computer Help Desk (223-HELP) 


253 


56.6 


26.1 


66 


21.7 


55 


52.2 


132 


Employee Benefits Office 


196 


43.8 


25.5 


50 


22.4 


44 


52.Q 


102 


ACCNet Services 


193 


43.2 


21.8 


42 


24.4 


47 


53.9 


104 


Staff Development Office 


181 


40.5 


19.3 


35 


16.6 


30 


64.1 


116 


Admissions and Records Office 


180 


40.3 


20.0 


36 


17.8 


32 


62.2 


112 


Bursar's Office (HBC) 


170 


38.0 


25.3 


43 


21.2 


36 


53.5 


91 


Office of Faculty Development 


168 


37.6 


14.9 


25 


20.8 


35 


64.3 


108 


Maintenance (Repairs) 


156 


34.9 


19.2 


30 


13.5 


21 


67.3 


105 


Employment Services 


150 


33.6 


46.0 


69 


20.0 


30 


34.0 


51 


Telephone Services 


149 


33.3 


16.1 


24 


17.4 


26 


66.4 


99 


Personnel Records Office 


149 


33.3 


34.2 


51 


17.4 


26 


48.3 


72 


Office of the AVP for Human Resources 


145 


32.4 


36.6 


53 


20.7 


30 


42.8 


62 


Purchasing Office 


140 


31.3 


26.4 


37 


19.3 


27 


54.3 


76 


Accounts Payable Services 


139 


31.1 


23.0 


32 


22.3 


31 


54.7 


76 


Central Warehouse Services 


130 


29.1 


11.5 


15 


15.4 


20 


73.1 


95 


General stores/inventory Services 


105 


23.5 


13.3 


14 


14.3 


15 


72.4 


76 


Employee Compensation Office 


104 


23.3 


45.2 


47 


16.3 


17 


38.5 


40 


Office of the VP for Business Services 


103 


23.0 


42.7 


44 


18.4 


19 


38.8 


40 


Office of the Dean of Arts & Humanities 


102 


22.8 


26.5 


27 


15.7 


16 


57.8 


59 


Office of the AVP for Information Technology 


97 


21.7 


16.5 


16 


23.7 


23 


59.8 


58 


IT Systems Support 


96 


21.5 


19.8 


19 


17.7 


17 


62.5 


60 


Office of the Dean of Math & Science 


95 


21.3 


21.1 


20 


12.6 


12 


66.3 


63 


HBC Room Scheduling 


91 


20.4 


16.5 


15 


19.8 


18 


63.7 


58 


Financial Aid Office 


87 


19.5 


10.3 


9 


13.8 


12 


75.9 


66 


Office of the EVP for Admin., Institutional Advance. & Com. Relations 


83 


18.6 


33.7 


28 


27.7 


23 


38.6 


32 


Office of the Dean of Soc. & Behavioral Sciences 


82 


18.3 


22.0 


18 


6.1 


5 


72.0 


59 


Workforce Training & Cont Education Office 


82 


18.3 


14.6 


12 


31.7 


26 


53.7 


44 


Physical Plant 


80 


17.9 


18.8 


15 


22.5 


18 


58.8 


47 


Office of the AVP, Institutional Effectiveness 


80 


17.9 


26.3 


21 


17.5 


14 


56.3 


45 
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tions/* Thus, the length of the survey may have ad- 
versely affected the response rate. Other comments 
suggested that some respondents were not familiar with 
the offices listed: ''The design of this questionnaire is poor 
at best. There should have been some kind of explanation of 
what each office was responsible for. / don 't believe that 
most employees know these departments 'official' name." 
This may have impacted response rates for individual 
offices. 

In the following analysis, results for the College- 
wide services are based on all respondents who indi- 
cated that they had requested or received the services of 
a specific office during the past year. Level of use is an 
important factor in analyzing employee satisfaction. If 
it can be assumed that offices that are used more have 
greater impact on employee satisfaction, then the Col- 
lege needs to focus its analysis on these offices. In the 
discussion that follows, all tables list College-wide of- 
fices/services in descending order of the number of re- 
spondents who reported they used the services of that 
office. Additionally, all percentages are of the total 
“user” responses to that item. And finally, responses of 
“satisfied” and “very satisfied” were aggregated into a 
single “satisfied” rating, as were responses of 
“dissatisfied” and “very dissatisfied.” 

Satisfaction with Overall Service 

Table 1 (page 2) displays the percentages of respon- 
dents who indicated they were satisfied, neutral, or dis- 
satisfied with the overall service provided by highly 
used offices. Of the highly used offices, the ones with 
the highest levels of satisfaction — at least 70% of re- 
spondents reporting they were satisfied with overall ser- 
vice — included Financial Aid Office (75.9%), Central 
Warehouse Services (73.1%), General Stores/Inventory 
Services (72.4%), and the Office of the Dean of Social 
and Behavioral Sciences (72.0%). Of course 70% satis- 
faction is an arbitrary standard; however, best practices 
would indicate that this standard is the minimum ac- 
ceptable if quality service is a goal of the office. 

Dissatisfaction 

To determine where the College needs to focus ef- 



forts on improving its services, levels of dissatisfaction 
must be analyzed. Again, referring to Table 1, of 30 
highly used offices, 18 are of special concern because 
over 20% of respondents indicated dissatisfaction with 
the services provided during the past year. The ser- 
vices of these offices warrant self-study and more de- 
tailed analysis. 

Respondents reported the highest levels of dissatis- 
faction with Overall Service of Employment Services 
(46.0%), Employee Compensation Office (45.2%), and 
the Office of the VP of Business Services (42.7%). 
For each of these offices, a greater percentage of re- 
spondents were dissatisfied with the overall services 
they requested or received than were satisfied. 

The discussion thus far has focused only on re- 
sponses for Overall Service. To assist in developing 

(Continued on page 4) 



Table 2; Comparison of Dissatisfaction with Promptness and 
Overall Service of Highly Used College-wide Offices/Services 


College-wide Office/Service 


Percent Dissatisfied 


Promptness 


Overall 

Service 


Difference 


Payroll Office 


35.1 


34.6 


0.5 


Computer Help Desk (223-HELP) 


33.5 


26.1 


7.4 


Employee Benefits Office 


33.8 


25.5 


8.3 


ACCNet Services 


28.6 


21.8 


6.8 


Staff Development Office 


21 .4 


19.3 


2.1 


Admissions and Records Office 


22.4 


20.0 


2.4 


Bursar’s Office (HBC) 


27.5 


25.3 


2.2 


Office of Faculty Development 


19.0 


14.9 


4.1 


Maintenance (Repairs) 


21.7 


19.2 


2.5 


Employment Services 


49.0 


46.0 


3.0 


Telephone Services 


22.7 


16.1 


6.6 


Personnel Records Office 


40.0 


34.2 


5.8 


Office of the AVP for Human Resources 


37.6 


36.6 


1.0 


Purchasing Office 


28.1 


26.4 


1.7 


Accounts Payable Services 


28.3 


23.0 


5.3 


Central Warehouse Services 


13.0 


11.5 


1.5 


General stores/inventory Services 


12.4 


13.3 


-0.9 


Employee Compensation Office 


44.4 


45.2 


-0.8 


Office of the VP for Business Services 


41.3 


42.7 


-1.4 


Office of the Dean of Arts & Humanities 


30.1 


26.5 


3.6 


Office of the AVP for Information Technology 


19.4 


16.5 


2.9 


IT Systems Support 


24.8 


19.8 


5.0 


Office of the Dean of Math & Science 


21.2 


21.1 


0.1 


HBC Room Scheduling 


17.4 


16.5 


0.9 


Financial Aid Office 


14.0 


10.3 


3.7 


Office of the EVP for Admin., Institutional 
Advance. & Com. Relations 


36.5 


33.7 


2.8 


Office of the Dean of Soc. & Behavioral Sci- 
ences 


20.2 


22.0 


-1.8 


Workforce Training & Cont Education Office 


18.3 


14.6 


3.7 


Physical Plant 


23.8 


18.8 


5.0 


Office of the AVP, Institutional Effectiveness 


29.1 


26.3 


2.8 
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an additional set of information, three dimensions of 
service were also rated: Promptness, Quality of Ser- 
vice, and Service Attitude. Trends in these dimen- 
sions may assist areas in developing plans to improve 
their effectiveness in meeting employee needs. 

In general, respondents were more dissatis- 
fied with Promptness than with Overall Service. As 
Table 2 (page 3) shows, for 20 (66.7%) of the 30 
highly used offices, at least two percent more respon- 
dents were dissatisfied with Promptness than with 
Overall Service. While Promptness tends to be prob- 
lematic for most areas, it is clearly an issue for sev- 



eral offices where at least five percent more respon- 
dents were dissatisfied with Promptness than with 
Overall Service. These include: Employee Benefits 
Office (8.3% difference), Computer Help Desk (7.4% 
difference), ACC Net Services (6.8% difference). 
Telephone Services (6.6% difference). Personnel Re- 
cords (5.8% difference). Accounts Payable (5.3% dif- 
ference), and IT Systems Support and Physical Plant 
(5.0% difference, each). 

On the other hand, dissatisfaction with Service At- 
titude and Quality of Service tended to track with dis- 
satisfaction with Overall Service (Table 3, below). 

( Continued on page 5) 



Table 3: Comparison of Dissatisfaction with Quality or Attitude to Dissatisfaction with Overall Service 

of Highly Used College-wide Offices/Services 


College-wide Offices/Services 


Dissatisfaction: 
Quality of Service 


Dissatisfaction; 
Service Attitude 


Dissatisfaction: 
Overali Service 


% 


difference 


% 


difference 


% 


Payroll Office 


34.0 


-0.6 


32.9 


-1.8 


34.6 


Computer Help Desk (223-HELP) 


24.2 


-1.9 


22.0 


-4.0 


26.1 


Employee Benefits Office 


28.6 


3.1 


23.9 


-1.7 


25.5 


ACCNet Services 


23.0 


1.2 


19.6 


-2.2 


21.8 


Staff Development Office 


20.8 


1.4 


18.0 


-1.3 


19.3 


Admissions and Records Office 


21.7 


1.7 


17.8 


-2.2 


20.0 


Bursar's Office (HBC) 


25.0 


-0.3 


27.2 


1.9 


25.3 


Office of Faculty Development 


14.8 


-0.1 


15.0 


0.1 


14.9 


Maintenance (Repairs) 


19.9 


0.6 


17.3 


-1.9 


19.2 


Employment Services 


46.1 


0.1 


39.9 


-6.1 


46.0 


Telephone Services 


16.0 


-0.1 


15.5 


-0.6 


16.1 


Personnel Records Office 


30.0 


-4.2 


32.5 


-1.8 


34.2 


Office of the AVP for Human Resources 


36.4 


-0.2 


31.3 


-5.3 


36.6 


Purchasing Office 


26.1 


-0.4 


21.1 


-5.3 


26.4 


Accounts Payable Services 


23.0 


0.0 


17.3 


-5.8 


23.0 


Central Warehouse Services 


10.9 


-0.7 


12.5 


1.0 


11.5 


General stores/inventory Services 


12.4 


-1.0 


15.2 


1.9 


13.3 


Employee Compensation Office 


43.7 


-1.5 


36.3 


-8.9 


45.2 


Office of the VP for Business Services 


39.8 


-2.9 


38.8 


-3.9 


42.7 


Office of the Dean of Arts & Humanities 


27.7 


1.3 


23.5 


-2.9 


26.5 


Office of the AVP for Information Technology 


17.3 


0.9 


15.3 


-1.2 


16.5 


IT Systems Support 


22.0 


2.2 


20.0 


0.2 


19.8 


Office of the Dean of Math & Science 


21.9 


0.8 


21.9 


0.8 


21.1 


HBC Room Scheduling 


16.5 


0.0 


18.7 


2.2 


16.5 


Financial Aid Office 


11.6 


1.3 


10.5 


0.1 


10.3 


Office of the EVP for Admin., Institutional Advance. & 
Com. Relations 


32.1 


-1.6 


32.9 


-0.8 


33.7 


Office of the Dean of Soc. & Behav. Sci. 


20.5 


-1.5 


22.9 


0.9 


22.0 


Workforce Training & Cont Educ Office 


16.0 


1.4 


18.3 


3.7 


14.6 


Physical Plant 


17.7 


-1.0 


16.7 


-2.1 


18.8 


Office of the AVP, Institutional Effectiveness 


22.5 


-3.8 


21.3 


-5.0 


26.3 
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Differences between dissatisfaction with Overall ser- 
vice and either dissatisfaction with Service Attitude 
or Quality of Services were less than two percentage 
points for most of the highly used College-wide Of- 
fices/Services. Exceptions were noted only for dissat- 
isfaction with Service Attitude of the Workforce 
Training and Continuing Education Office (3.7% dif- 
ference) and with Quality of Service of the Em- 
ployee Benefits Office (3.1% difference). 

Employee Group Trends 
For purposes of this analysis, employees were 
grouped into three categories: Full-time Faculty, Ad- 



Table 4: Ratings for "Overall Service" of Highly-used College-wide Offices/Services by Employee Category 


NOTE 1 : Adjunct Faculty includes Continuing Education and Adult Education Faculty. Non-faculty includes Administrators, 
Professional-Technical Staff, Classified Staff, and Hourly Staff. NOTE 2; “n” indicates the number of respondents. 

NOTE 3: “D” indicates dissatisfied, "N” indicates neutral, “S” indicates satisfied. 


College-wide Offices/Services 


Full-time Faculty 


Adjunct Faculty 


Non-faculty 


n 


%D 


%N 


%S 


n 


%D 


%N 


%S 


n 


%D 


%N 


%S 


Payroll Office 


60 


51.7 


25.0 


23.3 


69 


20.3 


23.2 


56.5 


129 


33.3 


24.0 


42.6 


Computer Help Desk (223-HELP) 


72 


34.7 


29.2 


36.1 


39 


33.3 


15.4 


51.3 


118 


18.6 


17.8 


63.6 


Employee Benefits Office 


52 


32.7 


15.4 


51.9 


20 


25.0 


45.0 


30.0 


109 


21.1 


21.1 


57.8 


ACCNet Services 


45 


40.0 


26.7 


33.3 


31 


35.5 


16.1 


48.4 


105 


14.3 


22.9 


62.9 


Staff Development Office 


38 


28.9 


18.4 


52.6 


39 


10.3 


23.1 


66.7 


83 


20.5 


12.0 


67.5 


Admissions and Records Office 


48 


27.1 


6.3 


66.7 


23 


13.0 


17.4 


69.6 


94 


17.0 


20.2 


62.8 


Bursar's Office (HBC) 


26 


23.1 


26.9 


50.0 


28 


32.1 


7.1 


60.7 


102 


24.5 


21.6 


53.9 


Office of Faculty Development 


60 


21.7 


20.0 


58.3 


54 


11.1 


24.1 


64.8 


43 


9.3 


20.9 


69.8 


Maintenance (Repairs) 


38 


28.9 


13.2 


57.9 


13 


38.5 


23.1 


38.5 


92 


9.8 


13.0 


77.2 


Employment Services 


41 


65.9 


17.1 


17.1 


26 


42.3 


19.2 


38.5 


71 


35.2 


22.5 


42.3 


Personnel Records Office 


32 


53.1 


12.5 


34.4 


22 


27.3 


18.2 


54.5 


81 


23.5 


19.8 


56.8 


Telephone Services 


31 


35.5 


35.5 


29.0 


12 


16.7 


33.3 


50.0 


85 


7.1 


12.9 


80.0 


Office of the AVP for Human Resources 


41 


51.2 


24.4 


24.4 


22 


31.8 


27.3 


40.9 


73 


28.8 


17.8 


53.4 


Purchasing Office 


35 


42.9 


11.4 


45.7 


3 


100.0 






89 


16.9 


22.5 


60.7 


Accounts Payable Services 


28 


46.4 


17.9 


35.7 


9 


33.3 


11.1 


55.6 


89 


13.5 


25.8 


60.7 


Central Warehouse Services 


23 


13.0 


13.0 


73.9 


2 






100.0 


93 


8.6 


17.2 


74.2 


General stores/inventory Services 


16 


12.5 


6.3 


81.3 


6 


50.0 




50.0 


71 


8.5 


16.9 


74.6 


Employee Compensation Office 


27 


59.3 


7.4 


33.3 


18 


44.4 


11.1 


44.4 


49 


32.7 


20.4 


46.9 


Office of the VP for Business Services 


34 


61.8 


17.6 


20.6 


6 


16.7 


66.7 


16.7 


55 


32.7 


20.0 


47.3 


Office of the Dean of Arts & Humanities 


31 


41.9 


12.9 


45.2 


17 


17.6 


5.9 


76.5 


45 


24.4 


17.8 


57.8 


Office of the AVP for Information Technology 


26 


23.1 


26.9 


50.0 


7 


42.9 


28.6 


28.6 


57 


10.5 


22.8 


66.7 


IT Systems Support 


19 


31.6 


21.1 


47.4 


5 


20.0 


20.0 


60.0 


64 


14.1 


21.9 


64.1 


Office of the Dean of Math & Science 


23 


43.5 


8.7 


47.8 


18 


22.2 




77.8 


45 


13.3 


20.0 


66.7 


HBC Room Scheduling 


27 


14.8 


18.5 


66.7 


6 


33.3 


16.7 


50.0 


50 


10.0 


22.0 


68.0 


Financial Aid Office 


21 


23.8 


14.3 


61.9 


1 






100.0 


58 


6.9 


13.8 


79.3 


Office of the EVP for Administration, Institutional 
Advancement & Community Relations 


22 


63.6 


22.7 


13.6 


11 


36.4 


27.3 


36.4 


43 


20.9 


30.2 


48.8 


Office of the Dean of Soc. & Behav. Sci. 


25 


28.0 


4.0 


68.0 


13 


15.4 


7.7 


76.9 


34 


23.5 


8.8 


67.6 


Workforce T raining & Cont Educ Office 


18 


16.7 


33.3 


50.0 


10 


20.0 


40.0 


40.0 


47 


10.6 


25.5 


63.8 


Office of the AVP Institutional Effectiveness 


33 


30.3 


12.1 


57.6 


7 


42.9 




57.1 


33 


18.2 


27.3 


54.5 


Physical Plant 


16 


37.5 


31.3 


31.3 


6 


33.3 


16.7 


50.0 


49 


10.2 


16.3 


73.5 



junct Faculty, and Non-faculty. Ratings of Overall 
Service provided by highly used offices are summa- 
rized in Table 4 (below) by each of these categories. 

Ratings varied by employee category. For exam- 
ple, while about half (51.7%) of Full-time faculty re- 
spondents were dissatisfied with the Overall Service of 
the Payroll Office, over half (56.5%) of Adjunct fac- 
ulty respondents were satisfied with the Overall Ser- 
vice of this office. 

Full-Time Faculty: A greater percentage of 
Full-time faculty were dissatisfied than were satisfied 

(Continued on page 6) 
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(Continued from page 5) 

with Overall Service of 11 (36.7%) highly used Col- 
lege-wide offices/services, while for 18 (60.0%) of 
these offices, a greater percentage were satisfied than 
dissatisfied. For one office/service (Computer Help 
Desk), Full-time faculty were equally (within 2.0%) 
satisfied as dissatisfied. Full-time faculty were most 
dissatisfied with Employment Services (65.9%), the 
Office of the EVP for Administration (63.6%), the Of- 
fice of the VP for Business Services (61.8%), Em- 
ployee Compensation Office (59.3%), Personnel Re- 
cords Office (53.1%), Payroll Office (51.7%), and the 
Office of the AVP for Human Resources (51.2%). 
They were most satisfied with General Stores/ 
Inventory (81.3) and Central Warehouse Services 
(73.9%). 

Adjunct faculty generally had lower levels of 
use than other groups. These respondents tended to be 
more satisfied than dissatisfied with the Overall Ser- 
vice of most (22 or 73.3%) College-wide offices/ 
services. However, a greater percentage were dissatis- 
fied than satisfied with the Overall Service of three 
(10.0%) college-wide offices/services. For five of- 
fices/services (16.6%) about an equal percentage 
(within 2%) of Adjunct faculty were satisfied as were 
dissatisfied with Overall Service. Adjunct faculty’s 
highest levels of dissatisfaction were with the Em- 
ployee Compensation Office (44.4%), Office of the 
AVP for Information Technology (42.9%), Office of 
the AVP for Institutional Effectiveness (42.9%), and 
Employment Services (42.3%). 

Non-faculty tended to be less dissatisfied in 
general than faculty groups. In fact, for all offices, a 
greater percentage of Non-faculty expressed satisfac- 
tion with Overall Service than expressed dissatisfac- 
tion. Non-faculty respondents’ highest levels of dis- 
satisfaction were reported for Overall Service of Em- 
ployment Services (35.2%), the Payroll Office 
(33.3%), Employee Compensation Office (32.7%) 
and the Office of the VP for Business Services 
(32.7%). 

Non-faculty were most satisfied with Overall Ser- 
vices of Telephone Services (80.0%), Financial Aid 
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Office (79.3%), Maintenance/Repairs (77.2%), Gen- 
eral Stores/Inventory Services (74.6%), Central Ware- 
house Services (74.2%), and Physical Plant (73.5%). 

Comments 

Written comments were made by a number of 
respondents. Most comments focused on areas relat- 
ing to customer service. Several mentioned customer 
service specifically: 

**All offices need to focus on customer service. ” 

**There are many departments at ACC that need cus- 
tomer service training. They also need to realize that we 
are each other* s customers. Sometimes I am saddened by 
the way some people treat co-workers and students. ** 

** However, my overall impression is that all units lack 
a sense of urgency. Services are provided when ACC em- 
ployee speed permits. ” 

**... employees are rude... ” 

Dissatisfaction with customer service was also 
reflected in comments about contacting offices. Many 
respondents referred specifically to difficulties related 
to the use of telephones: 

‘7 rated several offices dissatisfied because whenever / 
call them I can never get anyone to answer the phone and / 
am usually trying to help someone at that time. / think 
voicemail is overused. ** 

**An observation: when calling offices for service, there 
is a tendency on the part of clerk or office workers not to 
answer the telephone. The automatic answering machine 
does most of the phone answering. ” 

“A/y biggest complaint w/HBC is their reliance on 
voicemail. It is very hard to get hold of a live ’ person. 
Since adjuncts share offices and hold varied office hours, 
we can be hard to reach. " 

On the other hand, some respondents were 
more positive: 

‘7 am very, very excited to see the results of this. The 
majority of these services have improved a great deal in the 
past 6 mo! " 

‘7 have seen an improved, helpful attitude from several 
offices at HBC (this was not the case last year). ” 

‘7 love ACC; I love teaching there and also enjoy tak- 
ing classes . " 

— Martha Oburn, AVP 
Institutional Effectiveness 
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Using Rubrics to Assess Effectiveness 



Rubric is an odd term. Miriam defines it as “an 
authoritative rule” and “something under which a 
thing is classed.” It has come to mean some set of 
uniform criteria applied for the purpose of evaluation. 
For years teachers have applied rubrics, whether they 
used the term or not, to student work. “Your research 
paper must have a (1) cover sheet with the following 
information, (2) clear introductory, body, and con- 
cluding paragraphs, (3) at least 10 sources, 7 of which 
must be print, (4) and at least 15 footnotes,” an Eng- 
lish or History or Science teacher might say. Often a 
checklist is provided for the student to show which 
items were included, which missing. These are ru- 
brics. 

An on-line search shows rubrics used for hun- 
dreds of purposes, from assessing web pages to multi- 
media presentations, from oral reports to science labs, 
from research papers to teachers’ lesson plans. There 
are even rubrics to assess rubrics. 

Origin of Our Assessment: The ACC English 
Department began using rubrics to assess our students 
in Composition I and II classes several years ago as 
we were required to implement program evaluation. 
We identified purposes for our classes, set goals to 
determine if we were doing what we wanted in the 
classes, then measured whether we had reached the 
goals. With measurement we began applying rubrics. 

Origin of the Rubrics: How did we develop the 
rubrics we use? We identified content standards, de- 
ciding what we wanted to see in the Composition I 
and n exit exams. For example, the best papers (5 
points) include “rich content, varied sentence struc- 
ture, an appropriate tone, clear and appropriate organi- 
zation, very few and not serious mechanical errors, an 
opening that draws in the reader, a conclusion that 
summarizes and highlights the main points, an overall 
impression that the reader has been taught sentence 
after sentence, paragraph after paragraph.” Fair pa- 
pers (3 points) include “content which does not re- 
quire too many clarifications, occasionally varied and 
effective sentences, introductory paragraphs which 



provide generally clear main points, few serious me- 
chanical errors.” The worst papers (1 point) include 
“frequent and serious mechanical errors, evidence of 
scanty or no proofreading, only rudimentary organiza- 
tion, little content.” 

Process: Once each fall and spring semester, a 
number of English faculty meet on a Saturday morn- 
ing (coffee and sweet rolls help us remain inspired) to 
assess 100 “C” tests from Composition I and 100 from 
Composition II. Our syllabi are not as standardized as 
they once were but they still have some uniformity of 
requirements. The “C” tests, however, have the same 
format and requirements for all students in these 
classes and are usually administered in the testing cen- 
ters. We consider these as exit exams because the stu- 
dents must have completed all required papers to this 
point and then complete the “C” test in order to pass 
the class. They write additional papers for higher 
grades. 

We gather tests from all campuses and from as 
many different teachers as possible. We score only 
“C” tests which have been accepted by the instructor; 
that is, the students taking these tests have all passed 
the course with at least a “C.” 

In the evaluation session, before reading the tests, 
we review the rubrics and read and discuss a couple of 
sample papers. We consider this a recalibration. 
Most of us have done this before, but we need to re- 
mind ourselves, to be sure we are on the same track. 
Once we get to the actual papers for evaluation, each 
is read twice. The first reader evaluates the paper and 
marks it with a number (1 - 5) and covers the score so 
the second reader will not be influenced. After the 
second reading, the scores are compared. If they are 
no more than one score apart (reader one gave the pa- 
per a 4, reader two gave it a 3), the numbers are added 
and the paper is completed. If the two scores are far- 
ther apart than one point, a third reader evaluates the 
paper. The two higher scores are added. A combined 
score of 6 or higher is passing. Last April, 14 of us 
evaluated the 200 papers in about two hours. 

(Continued on page 8) 
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Recent Results; The department’s goal is to 
pass 70% of these tests with a 6 or higher. Remem- 
ber that all of the tests we read already have been 
passed by our English instructors. This spring we 
passed 69% of the Composition I tests (compared to 
75% in 1999, 65% in 1998). We passed 79% of the 
Composition II tests (73% in 1999, 63% in 1998). 

What We Do After: We realize that our being 
able to apply the rubrics and passing only 63 - 79% 
of the exit exams which have been 100% passed by 
others isn’t good enough. We need to address the 
inconsistency. Over the past several years, partly as 
a result of our assessment sessions, we have refined 
the syllabi and the “C” tests. We have reviewed all 
and changed some of the textbooks. We are work- 



ing on identifying benchmark papers so we can dis- 
tribute examples of what the department has agreed 
are typical of “A,” “B,” “C,” and “F’ papers. 

We believe that applying the rubrics helps us in 
determining what we need to do. The rubrics help 
establish a baseline. While it can be adjusted by 
changing the rubrics as we need to, the baseline es- 
tablishes clear criteria against which performance 
can be measured. The criteria are therefore less 
vague and the performance evaluations are more 
objective. 

— David Lydic, Professor 
English Department 



Unit-level Effectiveness Database Assessment Plan: English 

Academic Year: 1998-99 
Outcome Title: Composition II 

Intended Outcome: Students completing Composition II should be able to write a coherent essay of literary 
analysis identifying the elements of narrative fiction, using the appropriate methods of organization, and observing 
grammatical, mechanical, and stylistic conventions. 

Assessment Criteria: Through a process of collaborative holistic grading, 70% of Composition II students evalu- 
ated will have their final essays accepted. 

Methodology: Using English Task Force designed rubric, evaluate a sample of. 100 Composition II “C” tests 
from all campuses and from as many different teachers as possible 

Summary and Analysis of Data 

Results: Results were similar to last year. Once again, the department's 70% goal was not reached. Approximately 
65% of the exams in Composition I and II were passed with a combined score of 6; 10% of the exams required a 
third reader. 

Improvement Actions: The Communications Task Force has now begun the revision process for the Composition 
n syllabus. Faculty suggestions have been solicited, and a draft of the new syllabus should be forthcoming in 
Spring 1999. The Communications Task Force has begun review and selection of new stories and essays for the 
Composition II C tests with the goal of bringing the testing materials more closely into line with the goals of the 
courses. This should be completed by Fall 1998. The Communications Task Force continues to review the criteria 
for the holistic evaluation process. Holistic evaluation will of course continue, and the results will be distributed to 
all English faculty. 
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Using Pre and Post Tests to Assess Effectiveness 



Origin. 

After the last (Southern Association of Colleges 
and Schools (SACS) visit ACC was directed to adopt 
institutional effectiveness measures, both college-wide 
and at the unit level. The Government department be- 
gan by asking themselves what it was that they were 
trying to teach. They then asked how well they were 
teaching it. 

What do we teach? 

The Government Department began discussions 
regarding the common elements of both the United 
States Government and the Texas State and Local 
Government courses. In other words, what were those 
things that all Government instructors could agree that 
all students should know when they successfully 
passed the course. This proved to be an arduous task. 
Agreement was quickly reached on a number of learn- 
ing objectives, but others were more difficult. It also 
proved to be one thing to agree that all students should 
know “something” about the separation of powers for 
example, quite another to try to reach agreement on 
what that “something” should be. The concepts, 
terms, etc., that were agreed on were then made into 
learning objectives. 

How well are we teaching it? 

The learning objectives are distributed to all fac- 
ulty with the admonition that they are expected to 
teach them. The number of learning objectives have 
in the past been limited so that instructors would still 
have time to teach some topics of their own choosing 
and also allow for emphasis on a particular area or ar- 
eas. Each Fall semester, approximately 30 sections 
are randomly selected to participate in both pre and 
post tests of the learning objectives. It is also commu- 
nicated to faculty that if one or more of their sections 
are chosen, then they are expected to administer the 
test and return them to the Task Force. 

What are the results? 

The tests have demonstrated that students are 
achieving the learning objectives for the most part. Of 
course, some of the objectives have better success 
rates than others, but all in all, they do well. One 
somewhat puzzling aspect has been that there is often 
not a great deal of difference in success between pre- 
tests and post-tests. The reasons for this are not clear. 



It could be that we are teaching students what they al- 
ready know, to some extent, but we are re-enforcing it 
and broadening that knowledge. 

The same sections are not given both the pre and 
post test, reducing the validity of the test. The Task 
Force discussed at length whether an attempt should 
be made to pre and post test the same sections. How- 
ever, pre and post tests are best when they are admin- 
istered to the same students. Due to withdrawals, ab- 
sences, etc., testing the same sections would not nec- 
essarily result in the same individuals being tested 
both pre and post. The tests would also probably be 
skewed by the tendency for the better students to be 
left in a class at the end. As a result of these consid- 
erations both pre and post test sections have been ran- 
domly selected. 

Due to the logistical and validity problems associ- 
ated with pre and post testing the Government Task 
Force has assessed their assessment. In the Fall a 
common test will be administered by all faculty at the 
end of the semester incorporating a set of revised 
learning objectives for both United States Government 
and Texas State and Local Government. The tests 
will be required of all sections, but how it is incorpo- 
rated into the course is up to the instructor. In other 
words, the test could be part of a comprehensive final 
or a separate test, for example. It could be something 
that students study for or that they are surprised with. 
This methodology may raise more validity issues. 
Those findings would be part of next year's report. 

The process of assessment is not easy. As demon- 
strated above the Government Task Force has spent 
considerable time and energy attempting to assess 
their effectiveness. While the results may be less than 
ideal, the process itself has been worthwhile for at 
least two reasons. One, the debate over common 
learning objectives has made all of us who teach Gov- 
ernment rethink what we think is important for stu- 
dents to know. Secondly, it has made us keenly aware 
of the fact that it is not enough to merely give out in- 
formation. We must also make sure that the informa- 
tion is processed and internalized. These lessons are 
applicable to any discipline. 

— Rex Peebles, Professor 
Government Department 
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Unit-level Effectiveness Database Assessment Plan: Government 



Academic Year: 1 998-99 

Outcome Title: Knowledge of Basic Concepts and Processes 

Intended Outcome: Students who have completed a government course will possess a knowledge of the basic 
concepts and processes of politics and government. 

Assessment Criteria: A sample of students in randomly-selected government sections, including both required 
courses (GOV 2613 and GOV 2623) will take a pretest and a post-test which covers 30 central concepts and 
processes in politics and government. The students will answer at least 70 percent of the questions correctly and 
will improve one standard deviation (SD) from the pretest mean. 

Methodology: Tests will be administered during the fall semester, with a randomly-selected sample being tested 
during the first week of the semester and a randomly-selected sample being tested during the last week of the se- 
mester. 

Summary and Analysis of Data 

Results: Tests were administered in the Fall 1998 semester and were analyzed during the Spring 1999 semester. 
Government Assessment Results 1999 Assessment Results The scores on the post-test did not met the Task 
Force’s standards in terms of average or in terms of one standard deviation improvement. As the scores on the 
post-test demonstrate, the students who completed the courses were unable to average 70 percent correct answers 
in either course. Although students in GOV 2613 came closer to the 70 percent average stipulated by the Govern- 
ment Task Force than students in GOV 2623, they also began the course with a higher average on the pretest and 
presumably, more knowledge about the subject matter. Furthermore, students who completed the courses did not 
achieve an average score that was one standard deviation above the average of the students who had started the 
course. In addition to the descriptive statistics, two tests of samples were performed on the data: a Student's t-test 
and a Wilcoxon rank sum test or z-test. These statistical tests indicate whether statistically significant differences 
exist between the results of the two sets of scores (pretests and post-tests). Although both tests yield valid results 
in large samples, the Wilcoxon rank sum test is more appropriate for the data because its assumptions are less 
stringent. In both tests, the null hypothesis that there was no difference in the means was rejected. The results 
were significant at the .01 level of confidence . These tests indicate that the differences between the pretest re- 
sults and the post-test results were significant statistically. Although the students' scores did not achieve the Task 
Force's stipulated means, there were significant changes in the knowledge of the students after the completion of 
the courses. 

Improvement Actions: The Task Force developed more specific learning objectives for the key concepts for 
each course, GOV 2613 and GOV 2623, which will be included in the Government Instructors' Manual to be dis- 
tributed during the summer of 1999. Instructors are encouraged to distribute the learning objectives to their stu- 
dents and emphasize them in their teaching. 
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Using Test Pilot to Assess Effectiveness 



This year the Office of Institutional Effectiveness 
(OIE) conducted its annual stakeholder evaluation on- 
line. Test Pilot software by ClearLeaming, purchased 
by the College for use by faculty and staff, made the 
survey process easier, quicker, less expensive, and 
even enjoyable! It also improved our response rate 
over last year’s “mail-out” survey. 

Surveys, when carefully designed, are a useful 
tool for gathering both quantitative and qualitative 
data. However, they are also very time consuming. 
To get from the first discussion of “what do we want to 
know” to the final report of “this is what we found 
out” usually takes at least six months. Most of this 
time is spent printing and distributing the question- 
naires, then waiting for them to be returned, and finally 
transforming the responses into data that can be ana- 
lyzed. Test Pilot shortens this time. 

Last year, for our “mail-out” evaluation survey, we 
created questionnaire items that would give us infor- 
mation we needed to measure the effectiveness of the 
key missions of our office. This year, refining those 
items based on the analysis of data provided by the 
1999 evaluation, we created a Test Pilot questionnaire 
to be deployed (distributed) “on line” via email. That 
is to say, we sent an email message to all our stake- 
holders requesting they “click” on the link provided 
and complete the survey questionnaire on their 
browser. 

We started receiving responses immediately after 
the email went out; by the next morning we had al- 
ready received 86 responses; by the end of the second 
day, we had received 172 responses — 53 more than the 
total number of responses we received to the 1999 
“mail-out” evaluation! 

Submitted responses were automatically trans- 
formed into data that were entered into a table that I 
(as the “test owner”) could access on the web or 
download into an Excel spreadsheet. Descriptive sta- 
tistics for each item were calculated instantly and auto- 



matically, allowing us to see, at any point in time, the 
frequency and percentage of responses as well as the 
number of times a question was “offered” and not an- 
swered, the median value chosen, the mean of the re- 
sponses and the standard deviation. Reporting results 
to staff was as easy as printing the ‘Test Statistical 
Analysis.” 

The software is easy to use with “on-line” help that 
guides the author through the process of setting the 
“look and feel” of the survey (background color and 
style, headers and footers, text formatting and editing 
defaults), question design (yes/no, true/false, multiple 
choice, fill-in-the-blank, check all that apply, matching, 
Likert scale), and setting up security options. 

The security options are a key feature. As author of 
the assessment, an owner identification and password 
allow you to access response data. You can limit re- 
spondent access to the assessment via a list of “users” 
you include with the file, either by entering them indi- 
vidually, if there are only a few, to importing them 
from an Excel spreadsheet. You can limit access to the 
assessment by time or by the number of submissions 
made by a single user. You can restrict access by com- 
puter — that is by identification of specific computers 
that have access to the assessment. More security op- 
tions than probably anyone would need for conducting 
a survey. 

Once you have “authored” the assessment and it is 
ready for distribution, the file is uploaded to a web 
server and instantly ready for your respondents — 
perhaps graduates of your program, or their employers; 
or in the case of non-instructional units, your stake- 
holders/customers. 

For information on using Test Pilot to assess effec- 
tiveness in your area, contact me in the Office of Insti- 
tutional Effectiveness. 

— Roslyn Wallace, Coordinator 
Institutional Assessment 
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Defining effectiveness in terms of *'How well did we do?** (results) rather than ''What will we do?** (process) can 
present challenges to both instructional and non^ instructional units. But if gathering data for improvement is the 
reason for assessing effectiveness, this challenge must be addressed; the distinction must be made; and the indica- 
tors of effectiveness must be defined in terms of outcomes that identify intended results. In the following Q&A, 

Dr. Karen Nichols, Executive Director of Institutional Effectiveness Associates, addresses this issue. 



Q: How Does Assessment of Institutional Ef- 
fectiveness Relate to the Strategic Planning 
Already Taking Place on Many Campuses? 
On many campuses both strategic planning and in- 
stitutional effectiveness or assessment planning are 
conducted. They are necessary and often required by 
regional accrediting associations. However, they are 
different in their approach to planning and it is vitally 
important that institutions recognize and respect the 
different purposes which each serves (see Figure 3). 

Strategic planning on campuses is frequently a 
product of presidential leadership or the requirements 
of the governing board. Regional accrediting associa- 
tions also often require some form of strategic planning 
activity. Such planning focuses upon the question, 
"What actions should we take to implement the Ex- 
panded Statement of Institutional Purpose?" As such, 
strategic planning often results in a series of action 
plans (these are frequently long range) with resource 
requirements to put into action the necessary processes 
to accomplish the statement of purpose. Strategic plan- 
ning is characterized by: 

• Administrative planning 

• Fiscal planning 

• Physical facilities planning 

• Budget planning 

In AES [Administrative and Educational Support] 
units, such administrative planning often focuses on the 



means through which to improve processes or make 
the unit's operations more efficient. These adminis- 
trative planning efforts often include cross training of 
staff, improving communications with other adminis- 
trative offices, and implementation of improved re- 
cord-keeping activities. It is only after these adminis- 
trative planning activities have been accomplished 
and are part of the services provided by the unit, that 
they are subject to assessment regarding the extent to 
which they are effective in providing services to the 
unit's clients which is the end result sought. 

Strategic planning takes place in order for an in- 
stitution to survive and/or go about its development 
and refinement. A number of institutions have as- 
sumed that since strategic planning was being well 
accomplished on the campus, then institutional effec- 
tiveness was also being accomplished. The negative 
findings regarding effectiveness planning by regional 
accrediting association reaffirmation visitation com- 
mittees resulting from this assumption have been a 
shock to many of these institutions. 

Institutional effectiveness planning also relates to 
the statement of purpose for the institution. How- 
ever, it asks the basic question, "How well are our 
students learning and administrative (AES) services 
functioning?" Thus institutional effectiveness plan- 
ning is very ends or outcomes oriented and focuses 
upon the results of the institution's efforts (as opposed 

(Continued on page 13) 
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(Continued from page 12) 

to the efforts or processes implemented) as measured 
by services provided by AES units and student learn- 
ing in instructional programs. Institutional effective- 
ness planning is characterized by: 

• Expected results (administrative objec- 
tives) 

• Means of assessment 

• Actual assessment results 

• The use of results to improve services 

Q: If We’re Doing Strategic Planning, Isn’t 
That Enough? 

Both strategic planning and institutional effec- 
tiveness planning are necessary for a campus to be in 
compliance with most regional accreditation require- 
ments. The danger is that, because of the prior exis- 
tence of strategic planning activities, institutions of- 



ten do not implement institutional effectiveness plan- 
ning activities until after being scourged by their re- 
gional accrediting association. 

Certainly, the strategic action plans resulting 
from administrative and educational support unit ad- 
ministrative planning may cause an increase in re- 
quests for provision of resources with which to offer 
services (personnel, better equipment, office supplies, 
etc.) being requested or provided. However, if an in- 
stitution attempts to "assess" the accomplishment of 
its strategic action plans, it determines: 

• Were the personnel provided? 

• Was the equipment purchased? 

• Were there enough office supplies? 

• Was the new form implemented? 

(Continued on page 14) 




From The Department Head's Guide to Assessment Implementation in Administrative and Educational Support (AES) Units 

by Karen W. Nichols and James O. Nichols 
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(Continued from page 13) 

This focus is altogether insufficient and differs 
from the institutional effectiveness or ends focused 
assessment of the results of these activities or: 

• Measures of increased client satisfaction 

• Direct measures of increased levels of service 

• Validation of services by external reviews 

• Ability of the client after receiving service 

Units’ strategic action or administrative plans in- 
clude descriptions of services which the unit believes 
will be improved as a result of changes in an AES 
unit process. These administrative planning activities 
such as ’’implementing an on-line transcript service” 
or ’’designing an improved inventory check list” are 
found on the left side (strategic planning side) of the 
triangle shown in Figure 3. When the unit is prepar- 
ing administrative objectives for institutional effec- 
tiveness (found on the right side of the triangle shown 
in Figure 3), seldom do they include such administra- 
tive planning initiatives describing processes to be 
improved. 

While many of the regional accrediting associa- 
tions support the use of assessment results in institu- 
tional planning and budgeting, in the authors’ opin- 
ion, if assessment activities are identified widely on 
the campus as the means through which to justify or 
drive budget requests, the use of these assessment re- 
sults for service improvement will substantially di- 
minish. However, from a conceptual standpoint, use 
of assessment results to impact resource allocation 
and budgeting is very attractive. From a practical 
standpoint, it undermines the process of program or 
service improvement in several ways. First, if assess- 
ment results are seen primarily as justification for the 
request of additional resources, then the absence of 
additional resources will be widely utilized as a rea- 



son for not making service improvements. Under 
these circumstances, many AES units will never stop 
to consider what improvements could be made within 
existing unit resources, but will move directly to re- 
quests for additional resources which in many institu- 
tions stand little chance of funding. Second, in some 
AES units the means of assessment and criteria for 
success will be crafted in such a way as to justify a 
request for additional resources rather than improve 
services. Third, both of the previously described cir- 
cumstances will lead rapidly toward staff considera- 
tion of assessment as a means through which to ’’play 
the game,” rather than genuinely improve services 
offered. AES units are urgent to exercise caution in 
the use of assessment results as a justification for re- 
questing additional resources. Rather, AES units 
need to first ask, "Given the personnel and resources 
we currently have, how can our unit improve its ser- 
vices?" 

So, the answer to the question is simply, if you’re 
doing strategic planning which results in unit action 
plans, etc., that is probably not sufficient to meet re- 
quirements concerning institutional effectiveness and 
the improvement of services by educational and sup- 
port units. 

— Karen W. Nichols, Executive Director 
Institutional Effectiveness Associates 

This article was excerpted from The Department 
Head*s Guide to Assessment Implementation in Ad- 
ministrative and Educational Support Units with the 
generous permission of the author. Copies of the 
book are available from the Office of Institutional 
Effectiveness. 
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Graduate Outcomes — Workforce Detail 



To determine whether graduates of public commu- 
nity colleges in Texas are employed, or continuing their 
education, or both, the Texas Higher Education Coordi- 
nating Board (THECB) uses state employment records 
and various THECB data to locate students following 
graduation. However, these data may overlook self- 
employed graduates and graduates who have transferred 
to private or out-of-state schools. Additionally, the em- 
ployment records do not indicate whether the graduate is 
employed in the field for which the degree or certificate 
was granted. 

Each year, ACC surveys its graduates of the previ- 
ous academic year to provide data that supplement the 
THECB ’s tracking data. 

The ACC 1997-98 Graduate Survey was distributed 
during the Spring of 1999 to 1,033 graduates for the 
1997-98 academic year. A total of 454 (44%) question- 
naires were returned. Not all respondents answered every 
question in the survey. 

Graduates were asked to respond to questions about: 
■employment status after graduation 
■whether their degree or certificate was required 
for their job 

■any further education beyond their ACC work 
■any volunteer work involving ACC training 
■their opinion of how well ACC prepared them 
for employment or further education 



The chart below illustrates of the 1,033 graduates, 
405 (39.2%) are or were at one time after graduating 
employed. Of those, 357 graduates (34.1%) were 
employed in a job that was related to their ACC train- 
ing. Additionally, seven graduates (0.6%) volun- 
teered in an area related to their ACC training even 
though they did not use their training in a job. 

Most (370) of the graduates responding to the 
survey (445) rated their ACC preparation for employ- 
ment or further education “excellent” (46.5%) or 
“satisfactory” (35.0%). 



Please rate how well you feel ACC prepared 
you for employment or further education. My 





Respondents 
N % 


Excellent 


211 


46.5 


Satisfactory 


159 


35.0 


Good only in some areas 


60 


13.2 


Fair 


7 


1.5 


Inadequate 


8 


1.8 


Did not respond 


9 


2.0 



Follow-Up Survey Results; 1997-98 Graduates 

(Note: all percentages are of the total number of 1997-98 graduates.) 
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Unit-Level Effectiveness Documentation 

The Southern Association of Colleges and Schools (SACS) requires colleges it accredits to document their 
institutional effectiveness process. The ACC Unit-Level Effectiveness Database was designed to provide a 
centralized web-based venue where all College units document their institutional effectiveness process. At 
scheduled times throughout the academic year, as units complete the steps in the effectiveness process, information 
should be entered into the database to provide a complete record of the unit’s documentation, as required by 
SACS. 

Documentation Schedule: 

The unit enters its purpose statement into the database only once, unless it changes. For each effectiveness 
“loop”, it enters outcomes information at three other times to complete documentation of its institutional 
effectiveness process. 



What the unit enters: 


When the unit enters information into the database: 


Unit Purpose Statement 


Once for all academic years (unless it changes) 


Outcome Assessment Plan 


Fall semester of the current academic year 


Summary and Analysis of Data 


Summer semester of the current academic Year 


Impact Statement 


Summer of the following academic year 



Currently, units should be entering the Summary and Analysis of Data for each of the 1999-2000 outcomes 
they have assessed, the unit’s Impact Statement for improvements they made in 1998-99, and the Outcomes they 
will assess in 2000-01 . If you have any questions regarding what to enter, or how to enter it, contact 
Roslyn Wallace (223-7585 or rwall@austin.cc.tx.us). 

The database resides at http://salvador.austin.cc.tx.us/oiedb/intro.html 



jCAustin 
Community 

COLLEGE 



Richard Fonte, President Austin Community College 

Board of Trustees: 

Lillian Davis, Chair; Beverly Silas, Vice-Chair; John Worley, Secretary, 
Beverly Watts Davis, Allen Kaplan, Barbara Mink, 
Della May Moore, Carol Nasworthy, Rafael Quintanilla 

ACC is an equal opportunity institution. 

The ACC Effectiveness Review is published by 
the Office of Institutional Effectiveness at ACC. 

Roslyn Wallace, Editor 

OIE/HBC, Austin Community College, 5930 Middle Fiskville Road, Austin, TX 78752. 
Telephone (512) 223-7585 FAX (512) 223-7029 e-mail: rwall@austin.cc.tx.us 
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ULEAD — Assessing the Effectiveness of Effectiveness Assessment 
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Background 

The Southern Association of Colleges 
and Schools (SACS) requires the institu- 
tions it accredits to demonstrate the effec- 
tiveness of their educational programs. 
The Criteria for Accreditation, Section HI, 
Institutional Effectiveness, 3.1 Planning 
and Evaluation: Educational Programs 
states, 

"fhe institution must 

1. Establish a clearly defined purpose 
appropriate to collegiate education 

2. Formulate educational goals consis- 
tent with the institution's purpose 

3. Develop and implement procedures 
to evaluate the extent to which these 
educational goals are being 
achieved, and 

4. Use the results of these evaluations 
to improve educational programs, 
services, and operations. ” 

ACC’s effectiveness assessment proc- 
ess is guided by the answers to five criti- 
cal questions: 

1 . Why does this unit exist? 

2. What is the unit trying to accomplish? 

3. Did the unit succeed? 

4. What changes must be made? 

5. What difference did the changes make? 
These questions provide the framework 
for assessing whether the unit is “on 
track” or needs realignment of its proc- 
esses. Answering these questions com- 
pletes the five steps for designing and im- 
plementing an effectiveness assessment 
plan that yields specific information from 
which decisions may be made to improve 
the quality of student learning. 



Documenting effectiveness assess- 
ment plans serves three purposes: to 

demonstrate the College is meeting 
SACS requirements, to provide a record 
of individual units’ efforts to improve the 
quality of their programs, and to provide 
data for examining program trends. 

In Fall 1996, instructional units docu- 
mented their effectiveness assessment 
plans on paper forms. The plans were 
included in the report sent to SACS that 
year. The following year, the Office of 
Institutional Effectiveness created the 
Unit-Level Effectiveness Assessment Da- 
tabase (ULEAD) on the ACC Intranet to 
provide a centralized web-based venue 
where all College units would document 
their effectiveness assessment process. 

This article reviews the Effectiveness 
Assessment process at ACC and summa- 
rizes instructional unit-level assessment 
plans for academic years 1997-98 
through 2000-01 as documented in the 
ULEAD as of December 2000. 

(Continued on page 2) 
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Step 1 : Identify Purpose 

The first step in the ACC Effectiveness Assess- 
ment process is to identify the purpose of the unit in 
terms of the function it serves within the College. The 
purpose statement answers the question, “Why does 
this unit exist?” Each unit is an integral part of the 
College, and as such is responsible for a specific 
“task” that assists the College in fulfilling its state- 
mandated mission. 

A statement of the unit’s purpose includes identifi- 
cation of the people the unit serves (its stakeholders), 
what the unit provides those stakeholders, and the ma- 
jor benefit they may expect to realize from their asso- 
ciation with the unit. Stakeholders — those who will 
be affected by the unit — certainly include students, but 
may also include employers of graduates, four-year 
institutions, and even the community at large. 

As of December 2000, sixty (60) instructional 
units had documented a purpose statement into the 
ULEAD. 

Step 2: Identify Outcomes 

In step two of the Effectiveness Assessment proc- 
ess, the unit identifies the results it intends students to 
achieve and defines criteria it will use to determine 
whether those results have in fact been achieved. 
Learning outcomes are the vehicle for determining 
whether the unit’s purpose is being achieved. Out- 
comes information provides concrete, meaningful, and 
useful evidence of achievement of the unit’s purpose, 
i.e., whether the unit is doing what it proposes to be 
doing. Well-designed outcomes also provide informa- 
tion to direct improvement, if such is needed. Thus, a 
unit’s instructional outcomes serve as the foundation 
for assessment planning. 

Outcomes statements 

• provide direction to the program, 

• communicate what the faculty in the program 
deem important. 



• identify the intended results of the educational 
program, 

• describe what students should be able to know, 
think, or do, 

• describe the attitudes, values, and skills students 
will acquire by completing the educational 
program, and thus 

• are indicators of the effectiveness of a program. 

Outcome statements that describe instructional 
activities (goals) rather than student learning 
(outcomes) are not suitable for assessing a unit’s ef- 
fectiveness. Although goals and outcomes seem 
similar, some differences do exist. 



Goals 


Outcomes 


• broad, future-oriented 


• specific statements of meas- 


• reflect what the program Is 
striving towards or hoping to 
become 


• describe the desired 
learning/behaviors of stu- 
dents completing the pro- 


• refer to instructional proc- 


• refer to results of instruc- 


• used primarily in policy 
making and general program 
planning 


• used to assess effectiveness 
of curriculum and provide 
Information for improve- 



At a minimum, outcome statements should 
describe student behaviors and products that faculty 
would accept as evidence that learning was achieved, 
thus providing documentation of the effectiveness of 
the curriculum. Goal statements are helpful for direct- 
ing instructional activities, but are sometimes too 
general, broad, or vague for developing specific tools 
to assess student learning. Goals provide the context 
for outcomes. 

Outcomes statements are needed before 
appropriate assessment tools and procedures can be 
selected or designed. Having outcomes statements 

( Continued on page 3) 
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that are both clear and measurable greatly facilitates the 
identification of appropriate assessment techniques. 

Essential components of useful outcomes statements 
provide answers to the following questions: 

• Who will demonstrate the results of the unit’s 
achievement of its purpose? 

• When will the results be demonstrated? 

• What specific behavior or action will demonstrate 
these results? 

•What will students know or be able to do? 

•What skills will students have acquired? 

•What attitudes, values, or interests will 
students demonstrate? 

The table below summarizes instructional program 
outcomes statements for academic years 1997-98 
through 2000-01 as currently documented in the 
ULEAD. 

Many units use multiple outcome statements to as- 
sess the effectiveness of their unit. The most frequently 
listed intended outcome is that students demonstrate 
mastery of information. 



3. Assess Outcomes 

In step three of the Effectiveness Assessment 
process, the unit selects the methods it will use to 
gather information regarding the achievement of its 
intended outcomes, conducts the assessment, and re- 
ports the results. 

The methodology is the bridge between creating a 
plan to assess effectiveness and implementing that 
plan; it is the “recipe” for collecting the data. It de- 
scribes the actions that must be taken (and by whom) 
to determine whether the criteria for achieving the 
outcome have been met. Together, the criteria and 
methodology provide the means with which to ade- 
quately, accurately, and specifically document 
achievement of the outcomes. 

Two things are essential to keep in mind when de- 
signing methodologies: 1) assessment is concerned 

with the aggregate performance of a group of students 
rather than individual student performance and 2) 
you can’t fix in analysis what you bungled in design 
so design the methodology with an eye to how the 
data will be analyzed once it has been gathered. 



A variety of data collec- 
tion methods may be used. 
The methodology should 
include at least one direct 
measure of learning for each 
outcome. A direct indicator 
of learning is immediately 
observable. Such methods 
include pre- and post- 
testing, standardized exams, 
“common” final (or ques- 
tions on) exams, juried per- 
formances, etc. An indirect 
measure of learning may be 
included in the methodology 
as a second assessment tech- 
nique. Indirect indicators of 



Outcome Statement Programs 

Students demonstrate mastery of information in the field/course/program 32 

Students pass licensure/certification exams (external exams) 1 5 

Students complete the program/course 9 

Students are employed in their field 9 

Employers are satisfied with preparation/education of graduates 9 

Students are satisfied with the preparation they received for their career in the field 7 

Students who succeed in foundation course succeed in target course 6 

Students are employable 6 

Students demonstrate critical thinking/quantitative reasoning skills 4 

Students are satisfied with instruction/instructors 4 

Students are qualified to enter baccalaureate programs in the field 3 

Students succeed in baccalaureate programs 3 

Students demonstrate technical proficiency required for entry-level employment 3 

Students act in a professional/ethical manner 2 

Students pass TASP 1 

Students read at college level 1 

Students retain employment for one year 1 

Students are satisfied with the technical component of their education 1 

Students are satisfied with the learning environment re: values 1 

Students write effectively 1 



(Continued on page 4) 
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learning are subsequently observable. Indirect indica- 
tors might include survey information, job placement 
data, graduation rates, transfer studies, etc. 

The methodology should answer the following 
questions: 

• What “tool” will be used to collect the data? 

• When will the data be collected? 

• How will the data be analyzed? 

• How and to whom will the data be reported? 

•. Who will be responsible? 

According to documentation in the ULEAD, in- 
structional units use a wide variety of methodologies 
to gather information to assess whether they are 
achieving the outcomes they intended to achieve, in- 
cluding both direct and indirect measures and quantita- 
tive and qualitative data. 

• Performance on common (internal) exams 

• Performance on licensure/certification exams 

• Pre- and post tests 

• Comparison of student performance using and not 
using particular instructional software 

• Data from the student database 

• Surveys 

• Class presentations 

• Juried performances/products 

• Faculty evaluations 

• Course evaluations 

• External data (from the THECB) 

• Foundation/target assessment tests 

• Performance logs 

• Portfolios 

• Comparison of number of applicants to number of 
program completers 

• Comprehensive exit exams (internal) 

• Internship evaluations 

• Statistical comparisons of performance in founda- 
tion courses with performance in target courses 
(follow-up studies) 

• Aggregate holistic grading of random essays 

PAGE 4 



• Follow-up studies of students who transfer to bac- 
calaureate-granting institutions 

• Departmental records of enrollment and comple- 
tions 

Once the methodology has been implemented, the 
unit should communicate the findings to all faculty in 
the unit. The results need not be full-blown reports; 
they should be a summary of the findings, including 
the results of the assessment and the meaning the unit 
makes of those results. For example, it is sufficient to 
report, “Ninety percent (90%) responded they were 
satisfied. These results indicate employers rate our 
graduates qualified for their jobs.” or “The average 
was at the 48th percentile, indicating students are not 
retaining information from the foundation course.” 
Because the outcomes and criteria are stated in spe- 
cific terms, the results need only be reported in the 
terms specified in the criteria. 

The results should answer these questions: 

• What did the unit find out? 

• Were the criteria met? 

• What do these findings mean to the unit? 

• What problems need to be addressed? 

• What successes were identified? 

The ULEAD revealed many units have not docu- 
mented the results of their assessments. They have 
identified outcomes and methods for gathering data, 
but have not re-visited their assessment plans in the 
ULEAD to document the results they achieved. 

4 . I mplement I mprovements 

This step is where the assessment process comes 
full circle; where the assessment plan begins to “pay 
off.” If the outcome statement and criteria were clear, 
each result should indicate that the level of learning 
intended to take place either is or is not being 
achieved. This is also the step that SACS is particu- 
larly concerned about. 

( Continued on page 5) 
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The Improvement action plan must be specific 
enough that it can be implemented. It should answer 
the following questions: 

• What will the unit do regarding the assessed 
outcome? 

• What changes need to be made? 

• What other data are needed to make improve- 
ments? 

• What resources are needed to make improve- 
ments? 

• When will changes be made, and who will be 
responsible? 

If the findings indicate the outcome criteria have 
been met or surpassed, the unit may decide that no 
change is needed and report “Outcome will be as- 
sessed once more next year to validate results.” Then 
when the outcome is assessed again and the results are 
repeated, the unit should move on to assess another 
outcome. 

On the other hand, if the findings indicate the out- 
come criteria have not been met, the results of the as- 
sessment must be examined further. 

For example, the results may, in the judgment of 
the faculty, indicate a weakness in the unit in a very 
specific area where changing X will correct the prob- 
lem. Assuming change X is implemented, the unit 
would report, “Change X undertaken.” The outcome 
would be assessed again the following year using the 
same methodology to discover whether the change im- 
proved achievement. 

Another example, however, would be that the re- 
sults indicate a weakness, but it is not obvious where 
or what change is needed. In this case, the improve- 
ment plan might be to establish a committee to exam- 
ine the situation and report at a later date. The unit 
would report something along the lines of, 
“Committee appointed to review W.” Once the com- 
mittee made its report (before the end of the next as- 



sessment cycle) a more detailed improvement plan 
would be drafted, and the outcome would be assessed 
again in the next assessment cycle. 

If after a second and third assessment cycle the 
unit is still not achieving the outcomes it expects, it is 
time to examine the outcome (to see if it is realistic), 
the criteria (to see if it is reasonable), and the method- 
ology (to see if it is reliable). 

As with the documentation of results, many in- 
structional units have not documented their improve- 
ment action plans into the ULEAD. Of those who had 
documented improvement plans in the ULEAD, many 
indicated, when outcomes were not achieved and 
changes were needed, they made changes to curricu- 
lum, to instructional methodologies, and to tools used 
in the effectiveness assessment plan. Additionally, 
some units changed procedures they used for tracking 
graduates, policies, relationships with advisory com- 
mittees or the community, or assignment schedules. 

In some units, where the outcomes criteria were 
achieved, improvements were still initiated because 
the results of the assessment provided serendipitous 
improvement information — insights into connections 
between instruction and student performance. Other 
units’ improvement plans, in response to achieving 
the intended outcome, were to assess a different out- 
come the next academic year. A few units were satis- 
fied with the results of their assessments and indicated 
no improvements were needed. 

5. Identify Impact 

This is the “dessert” of the assessment process. 
The unit looks back to see how far it has come in its 
quest for quality. The impact statement makes the 
value of the assessment process visible — not only to 
the unit, but to everyone. 

At the beginning of each effectiveness assessment 
cycle, the unit needs to review its assessment plan 
from the previous year and consider the changes that 

(Continued on page 6) 
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have occurred as a result of the improvements it made. 
The impact statement answers the question, ‘What 
difference(s) did the improvement actions taken by the 
unit make?” 

If the unit changed curriculum, the impact state- 
ment is simply a statement of how that has affected 
student or faculty performance. Sometimes, however, 
there are unexpected and presumably unrelated 
changes as a result of improvement actions. For ex- 
ample, if an improvement action included surveying 
employers for input into skills in which they wished 
program graduates were niore proficient, an unex- 
pected impact of implementing that improvement 
might be closer relationships with those employers in 
other ways as well. 

Documentation of the impact of the improvement 
actions should take place a year following the identifi- 
cation of the implementation of the improvement ac- 
tions. 

Originally, the ULEAD did not allow units to 
document the impact of their improvement actions; 
this capability was added to the database in Fall 1998. 
As of December 2000, ten instructional units have 
documented the impact of their improvement actions 
into the ULEAD. The major effects of improvement 
actions these instructional units implemented include 
the following: 

• Improved student planning 

• More concise and relevant student advising 

• Increased certification exam pass rates 

• More graduates because more take license 
exam 

• Improved curriculum 

• Improved assessment methodology 

• Confirmation of quality of curriculum 

• Graduates better prepared for employment 

• Maintained enrollments 

• Program reputation “good”- attracts students 



Conclusion 

The Unit-Level Effectiveness Assessment Data- 
base is a useful tool for assessing the effectiveness of 
the unit-level effectiveness assessment process at 
ACC. It is also a useful tool for demonstrating com- 
pliance with SACS criteria 3.1. However, for that tool 
to provide “adequate, accurate, and specific” data, all 
instructional units must participate in its use to docu- 
ment their effectiveness assessment plans. Addition- 
ally, all five effectiveness assessment process steps 
must be documented. 

— Roslyn Wallace, Coordinator 
Institutional Assessment 



ULEAD Documentation 



• Database resides at 

http://www2.austin.cc.tx. us Voiepub/unitlevel.htm 

• User name: ACC Units (two words) 

• Password; unitgoals (one word) 

• Deadline; June 1 5 of each year; however 
documentation may be entered into the data- 
base at almost any time during the year. The 
database will be closed to data entry from June 
15 through July 15 to allow the Office of Insti- 
tutional Effectiveness to run reports on the 
status of unit-level documentation. 

• Unit-Level Effectiveness Assessment Database 
Manual available on-line at 

http://www2.austin.cc.tx. us. /oiepub/unitlevel.htm 

• Contact Suzanne Lucignani (223-7603, 
slucigna@austin.cc.tx.us) if you encounter 
technical problems during the data entry proc- 
ess. 

• Contact Roslyn Wallace (223-7585, 
rwall@austin.cc.tx.us) if you need assistance 
with, or have questions about, content. 
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Using ULEAD — Data Entry 

Getting to the database: 

1 . Open your internet browser (e.g., Internet Explorer 
or Netscape Navigator). 

2. Go to ACC internet home page (www.austin.cc.tx.us). 

3. Scroll down, click on ACC Intranet Home Page (for 
Faculty and Staff). 

4. Click on Dr. Font6, President. 

5. Click on Institutional Effectiveness. 

6. Under green tack, click on Unit-Level Effectiveness Da- 
tabase. 

7. Click on Unit-Level Effectiveness Database again. 

8. A message box appears: You are leaving the OIE Inter- 
net site and entering a password-protected area. If you do 
not have a password, please contact Roslyn Wallace at 
(512) 223-7585 or rwall@austin.cc.tx.us. Click OK. 

9. You are now at the ULEAD home page. It is a 
good idea to bookmark this address. From this 
page you may create, view, or edit unit purpose 
statements and effectiveness assessment plans. 

Create Function: 

To document the unit’s purpose statement or assess- 
ment plan into the ULEAD, you will use the Create 
function. 

1 . From the ULEAD home page, click on Create a new 
Unit Purpose Record or Create a New Outcome Assess- 
ment Plan. 

2. Type the requested information in each text box. 

If you need help with what to enter, click on the 
name line for that text box and a help screen will 
guide you. 

3. When you have completed typing the information 
into the text boxes, click on the Submit button at the 
bottom of the screen. You will be asked to provide 
your User Name and Password. 

4. In the User Name box, type ACC Units, then press 
the tab key on your keyboard. 

5. In the Password box, type unitgoals, then click on 
the OK button. 

6. Your record, whether a purpose statement or an as- 
sessment plan, will have been added to the data- 
base. PRINT a copy for your records by clicking 
on the print icon of your browser or by selecting 
print from the file menu across the top of the 
screen. 

It is not necessary to complete a record in one data en- 
try session, so long as you submit what you have com- 
pleted into the database by clicking on the submit but- 



ton at the end of the form. You may go back and add 
more information at a later time by using the search 
function. 

View/Edit Function: 

To view or edit documentation in the database, re- 
quires the use of a search page to find the specific re- 
cord you wish to either view or edit. From the 
ULEAD homepage, 

1 . Click on View/Edit an existing Unit Purpose Record or 
click on View/Edit a Unit Outcome record depending 
on whether you are working on the purpose or the 
outcomes. 

2. In the Search Unit window, click on the ^ 

3. Scroll down to find your unit’s name from the 
drop down menu. Click on it to select it. 

4. If you wish to sort the results of your search, by 
year or other criteria, click on the ^ in the Sort 
By window. Select the sort criteria you want to 
use and click on it. 

5. Click on the Start Search button. 

6. On the Search Results page, click on the unit code 
in the unit (first) column to view or edit the par- 
ticular record you want. 

7. To edit the record, scroll down and click Edit Re- 
cord or Edit this Outcome. 

8. In the appropriate field (box) type in the edits or 
changes you wish to make. (If you need help, 
click on the underlined name to the left of the field 
or window in which you are working.) 

9. Scroll down to the bottom of the form and click 
Submit Changes. 

10. On the Enter Network Password box, in the User 
Name field, type in ACC Units and in the Password 
field, type in unitgoals. Click OK. The edited 
document will be entered into the database. 

11. To return to the ULEAD homepage, scroll down 
to the bottom of the document and click Return to 
ULEA Database Home Page. To exit the database, 
click X in the top right comer of the window. 

Help with data entry (including screen shots) can 
be found in the ULEA Database Procedures 
Manual online at http://www2.austin.cc. tx.us./oiepub/ 
unitlevel.htm or contact Roslyn Wallace at 223-7585 or 
rwall@austin.cc.tx.us. 
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The Practical Why 

In its 1998 Criteria for Accreditation, the 
Southern Association of Colleges and Schools 
(SACS) mandates that associate programs 
(leading to Associate of Arts, Associate of Sci- 
ence, and Associate of Applied Science degrees) 
must require at least 15 semester hours of general 
education courses, including "at least one course 
from each of the following areas: humanities/fine 
arts, social/behavioral sciences, and natural sci- 
ences/mathematics." The institution "must dem- 
onstrate that its graduates ... are competent in 
reading, writing, oral communication, fundamen- 
tal mathematical skills and the basic use of com- 
puters" (17). ACC has instituted the 15-hour re- 
quirement, has specified seven general education 
outcomes we expect our students to be able to 
demonstrate, and has identified about 90 courses 
we say will help instill the skills and knowledge 
implied by SACS. 

The Real Why 

So what? All this does nothing to explain the 
WHY of general education. 

While we do not want to reduce intellectual 
pursuit to a list of skills and courses, the assump- 
tion of general education is that a college can craft 
a purposeful and coherent curriculum to help its 
students understand more and think better. In his 
article "Why Do I have to Take This Course: 
Credit Hours, Transfer, and Curricular Coher- 
ence," appearing in the Winter 2000 issue of 
AACU's Peer Review, Robert Sheonbereg says 
that the questions we should ask ourselves about 
any curriculum include "[what are the] education 
intentions of the courses and the connections 
among them? Do they cohere in the minds of in- 
dividual professors and students? When added 
together, do they comprise a meaningful 
whole?" (5). 

ACC has asked these questions. One answer 
has been to identify general education courses and 



to create the following seven outcomes that should 
help gauge whether our students have "acquir[ed] 
the knowledge and skills to prepare for a career, 
further their educational study, and contribute to so- 
ciety" (2000 - 2001 ACC Catalog 35). The general 
education courses are designed to 
"help associate degree students to: 

1. Possess sufficient literacy skills of writing, 
reading, speaking, and listening to commu- 
nicate effectively above the 12*'’ grade level, 

2. Understand numerical data and their impli- 
cations for daily living, 

3. Possess consciousness of society, 

4. Think and analyze at a critical level, 

5. Appreciate multi-cultural, multi-ethnic con- 
tributions to our country, 

6. Understand our technological society, 

7. Possess basic skills in the use of computers” 

We have, in Schoenberg's words, developed 
"system-wide goals for general education." If we 
are to "gain broad faculty and student understand- 
ing of them," we must "come up with ways to as- 
sess them." We are doing that now. 

Assessment 

The General Education Review Committee was 
formed by Dr. Fonte in spring 2000 and has been 
meeting all summer and into the fall, researching, 
discussing, and defining how we should assess gen- 
eral education at ACC. On the one hand, the gen- 
eral education courses obviously should be a focus. 
Are the courses doing what we say they do? On the 
other hand, the general education outcome state- 
ments imply skills and knowledge beyond the con- 
fines of specific courses, so we should assess more 
broadly as well. For example, writing is taught in 
Composition I and II, but the skills may be demon- 
strated in Psychology or History or Nursing. The 
committee will assess both general education and 
other courses. 

How will we assess? We will use a process 
called the Institutional Portfolio, first developed by 
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( Continued from page 8) 

Johnson County Community College in Overland 
Park, Kansas. Five ACC faculty committees 
(totaling about 57 teachers from about 35 different 
disciplines) are currently working to articulate gen- 
eral education outcome statements more measurable 
than the seven listed in the catalog. The committees 
are also developing scoring criteria and will set the 
target levels to determine whether outcomes have 
been met. The committees will become scoring 
teams in the spring, applying the criteria to student 
work. 

Each semester faculty will be asked to provide 
student work that will be “scored” using the scoring 
rubrics developed by the committees for each gen- 
eral education outcome. Scores will be aggregated 
for each outcome to give us a data-based picture of 
the performance of our students relative to the gen- 
eral education outcomes we are intending students 
to achieve. 

Will other faculty be involved? Yes. This 
spring all faculty will have a chance to give feed- 
back on the committees' preliminary work. Once 
that work has been finalized, the measurable out- 
comes and scoring criteria will be distributed to all 
faculty, who will be asked to supply student work. 

Basically, all faculty will be asked, "Do you re- 
quire any assignments that you think meet one or 
more of these outcomes?" Some of the faculty 
members responding will be asked to submit an en- 
tire section's worth of ungraded assignments. Either 
the faculty member will copy the assignments and 
submit them or a member of one of the general edu- 
cation committees will copy the assignments and 
return the originals to the teacher as soon as possi- 
ble. The committees will score the samples. Our 
goal is to score two hundred student samples for 
each general education outcome each semester. 

How will the results be used? First, the assess- 
ment results will be reported to the entire college. 
The primary purpose of the assessment is to begin a 
conversation among faculty. We will wonder why 
certain outcomes have not been met. We will scru- 



tinize and perhaps modify curriculum. In short, we 
will keep a sharper eye on what we do and how it 
works. 

Conclusion 

Assessment of general education this year can 
be seen as preliminary to the larger self-study we 
are beginning for SACS reaccredidation. Assess- 
ment of general education, however, is the start of a 
continuous process. This spring is a pilot assess- 
ment for all of us. We are testing the waters of our 
research methodology, of faculty involvement and 
response, of our own committee infrastructure, and 
of how the data are analyzed and reported. We will 
see what we learn and modify as necessary. 

Even if the shadow of SACS were not over us, it 
is simply a good idea to see if we are doing what we 
say we are. 

— David Lydic, Interim Dean 
Arts and Humanities 

For Your Information 

The General Education Review Steering Committee has a 
webpage at http://www2.austin.cc.tx.us/gened/. On this site 
are agendas and minutes of our meetings, a more complete de- 
scription of the Institutional Portfolio process, a complete list 
of subcommittee membership, and additional documents sup- 
porting the committee's work. As soon as possible, we will 
post the subcommittees' work on outcome statements and scor- 
ing criteria to the site. Feel free to contact any of us with ques- 
tions or comments. 



Name 


E-mail address 
....@austin.cc.tx.us 


Subcommittee 


David Lydic 


lydic 


Steering 

Committee Chair 


Anne Dunn 


adunn 


Critical Thinking 


Mary Parker 
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Mathematics 


Rex Peebles* 


Peebles 


Social/Cultural 

Awareness 


Lennis Polnac 


1 polnac 


Communications 


Julie Todaro 


j todaro 


Technological 
Awareness/ 
Computer Literacy 


Jane Latham 
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Roslyn Wallace 
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*New Chair of Steering Committee. Dr. Lydic has accepted 
appointment as Interim Dean, Arts & Humanities. 
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ACC Effectiveness Review 

Academic Programs Transfer Effectiveness Measures Trend Data 

Austin Community College, in its mission statement (Board Policy A-1 [b]), states that it provides “Freshman- 
and sophomore-level academic courses leading to an associate degree or serving as the base of a baccalaureate de- 
gree program at a four-year institution.” To determine whether the College is achieving this mission, an effective- 
ness assessment plan was developed by the President’s Effectiveness Council (PEC). Each year, the plan is imple- 
mented and the results are presented to the Board of Trustees and published on the Office of Institutional Effec- 
tiveness (OIE) web site. Following is one outcome of the effectiveness assessment plan for the transfer function 
and a chart displaying results for the last six First Time in College (FTIC) cohorts. 



Purpose: To prepare students for successful college or university transfer. 

Outcome/Criterion: ACC’s transfer rate will be higher than the statewide transfer rate for First Time in College 
(FTIC) students who have earned at least 15 semester credit hours (SCH) in baccalaureate transfer courses and 
who transfer to a four-year college or university within four years of their initial enrollment at ACC. 

Methodology: Each year the Coordinator of Institutional Assessment will compare ACC and Statewide transfer 
rates as published in the Texas Higher Education Coordinating Board (THECB) Community College Transfer Rate 
Study. 

Results: 

Comparison data for ACC and statewide transfer rates for cohorts from 1990 through 1995 indicate the following: 

• ACC’s transfer rate has been consistently higher than the statewide rate 

• The difference between ACC’s rate and the statewide rate has ranged from 0.2% greater for the 1994 
cohort to 1 1.3% greater for the 1991 cohort. 

• ACC’s transfer rate has generally increased over time. However, so has the statewide rate. 

• Just over one third of ACC students majoring in academic programs and just under one third of stu- 
dents statewide majoring in academic programs transfer . 



Comparison of ACC and Statewide Transfer Rates 
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Workforce Graduates Detail 

To determine whether graduates of public commu- 
nity colleges in Texas are employed, or continuing their 
education, or both, the Texas Higher Education Coordi- 
nating Board (THECB) uses state employment records 
and various THECB data to locate students following 
graduation. However, these data may overlook self- 
employed graduates and graduates who have transferred 
to private or out-of-state schools. Additionally, the em- 
ployment records do not indicate whether the graduate is 
employed in the field for which the degree or certificate 
was granted. 

Each year, ACC surveys its graduates of the previ- 
ous academic year to provide data that supplement the 
THECB ’s tracking data. 

The ACC 1998-99 Graduate Survey was distributed 
during the Spring of 2000 to 1025 graduates for the 
1998-99 academic year. A total of 340 (33.2%) question- 
naires were returned. Not all respondents answered every 
question in the survey. 

Graduates were asked to respond to questions about: 
■employment status after graduation 
■whether their degree or certificate was required for 
their job 

■any further education beyond their ACC work 
■any volunteer work involving ACC training 
■their opinion of how well ACC prepared them for 
employment or further education 
The chart below illustrates, of the 340 graduates re- 
sponding to the Graduate Follow-Up survey, 285 
(83.8%) are, or were at one time after graduating, em- 



ployed. Of those, 227 graduates (79.6%) were em- 
ployed in a Job that was related to their ACC training. 
Of the 43 respondents who did not use their training 
in their Job, three (7.0%) volunteered in an area re- 
lated to their ACC training. Additionally, of the 55 
respondents who reported they were not employed, 
five (9.1%) used their ACC training in volunteer 
work. Thus, of the 340 graduates responding to the 
survey, 235 (69.1%) used their ACC training either in 
their Job or volunteer work. 

Most (289) of the graduates responding to the 
survey (340) rated their ACC preparation for employ- 
ment or further education “excellent” (46.5%) or 
“satisfactory” (35.0%). 



Please rate how well you feel ACC prepared you 
for employment or further education. My 
preparation was... 




Respondents 




N 


% 


Excellent 


159 


46.8 


Satisfactory 


130 


38.2 


Good only in some areas 


32 


9.4 


Fair 


8 


2.4 


Inadequate 


5 


1.5 


Did not respond 


6 


1.7 


Total 


340 


100.0 



Follow-Up Survey Results: 1 998-99 Graduates 
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Qj I understand that we need to document that we 
are assessing the effectiveness of our units, but who 
is supposed to be responsible for making sure the 
documentation is in the Unit-Level Effectiveness 
Assessment Database (ULEAD)? 

Effective this spring, for instructional units, the Dean 
is ultimately responsible for making sure the units un- 
der his/her administration have documented their as- 
sessment plans into the ULEAD. Task Force Chairs 
and Program Coordinators are responsible at the unit 
level for making sure documentation is completed. 

Qj The methodology for our unit’s assessment plan 
clearly stated the Office of Institutional Effective- 
ness (OIE) would provide data to us. We still have 
not received any data from them. What gives? 

The Office of Institutional Effectiveness (OIE) can 
provide customized data for individual units, and does 
so quite often. However, to receive customized data 
from OIE, someone must let OIE know (via a data re- 
quest) what data are wanted — the documentation in 
ULEAD is not a data request. Data request forms are 
available on the OIE web site (accweb.austin.ee. tx.us/oie/ 
datareq.htm), or you can email the office at 
oieinfo@austin.cc.tx.us, or call 223-7601. 

Qj I looked in the ULEAD for the assessment plan 
our unit did as part of Program Review and it was 
not there. What happened? 

Those units undergoing program review are required 
to create an assessment plan for their program. It is 
included in the program’s self study report. Docu- 
mentation of those plans, however must be entered 
into the ULEAD by someone from the program/unit. 




0/ Our unit surveyed graduates to see if they were 
employed in the field. Problem was, out of the ten 
graduates we sent surveys to, only one graduate re- 
sponded. A lot of the forms were returned due to 
bad addresses. The question is, do we count the 
one response as 100% (since it was the only one we 
got and the graduate was employed) or as 10% em- 
ployment rate (since we sent out 10 forms and got 
back only 1)? 

It depends. First of all, look back at the outcome and 
criteria you defined. If they said something along the 
lines of “...of the graduates responding to a survey...” 
then you may use the 100 percent; however if the out- 
come and criteria said “..of the graduates X% will re- 
port they are employed....” you must use the 10 per- 
cent. 

In either case, you may wish to think again about the 
methodology you have chosen to assess this outcome 
since it does not appear to give you information you 
can use to make program decisions. A different meth- 
odology might include contacting area employers in 
your field to track your graduates (this is where advi- 
sory committees come in handy) or asking graduating 
students (perhaps those in the “capstone” course) 
whether they will be taking a job in the next few 
months. Your improvement plan would include defin- 
ing a new methodology. 
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A ssessing General Education Outcomes 
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Background 

Many faculty will recall that the Southern 

Association of Colleges and Schools ^ _ 

(SACS) and the Texas Higher Educa-'j ' • 
tion Coordinating Board (THECB) are 
emphasizing continuous improvement s 
through assessment in all areas of the 



college. ACC has established seven; 



general education student learning out- 
comes which should help gauge 
whether our students have acquired the ■ 
knowledge and skills to prepare for a 
career, further their educational study, 
and contribute to society. According to ; ; 
the 2000-2001 ACC Catalog, 
“associate degree students should 
1. possess sufficient literacy skills of 
writing, reading, speaking, and listen- 
ing to communicate effectively above " 
the grade level, 
understand numerical data and their 
implications for daily living, 
possess consciousness of society, 
think and analyze at a critical level, » 
appreciate multi-cultural, multi-ethnic . 
contributions to our country, ^ 

understand our technological society, ; 
and 

possess basic skills in the use of computers.” 



J{ave our 
graduates 
acquired the 
“hnowhedge and 
shifts to -prepare 
fora career, 
further their 
educationaC 
study, and 
contribute to 



society? ” 

ACC Catatog ^ 



The General Education Review Commit- 
tee has been at work for over a year 
'I studying and discussing ways to assess 
■ i whether ACC’s associate degree stu- 
; dents are achieving these outcomes. 
I' We have reviewed materials from vari- 
i;,;: ous other colleges that have instituted 
assessment of their general education 
outcomes. In each case, the school 
: made its outcomes specific enough to 
be assessed and developed a method 
" for assessing them. After much discus- 
sion and review, the committee de- 
cided to use an institutional portfolio 
process to document students' achieve- 
ment of general education learning out- 
; comes. 

The institutional portfolio process 
examines existing student "artifacts" 
from various disciplines and scores 
them using "rubrics" designed to estab- 
lish achievement criteria for each gen- 
eral education outcome. Artifacts are 
student work such as papers, exercises, 
videotaped .presentations, etc. that are 

(Continued on page 2) 
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assigned, collected, and graded by instructors in 
the normal conduct of a course. Rubrics provide 
a common set of standards for each general edu- 
cation outcome, regardless of the course from 
which the artifact was assigned and regardless of 
the grade the student earned from the instructor. 

The issue of gathering the artifacts of all stu- 
dents or just those of graduates was also discussed 
at some length. The committee decided that it is 
simply impractical to examine the work of gradu- 
ates only. First of all, in transfer programs, which 
make up seventy percent of the college’s courses 
and house all of the general education courses, 
graduates are few in number compared to the 
overall number of students. Secondly, by defini- 
tion, graduates have left the college, making it 
difficult to identify graduating students at the 
front end of the process. However, since we will 
be attaching student IDs to the student artifacts 
when collected, the total number of hours com- 
pleted by each student can be determined. We 
will then have a basis for comparing student per- 
formance with the number of hours completed to 
determine if progress has been made over time. 

Update 

There are five subcommittees of the General 
Education Review Committee: Communications, 
Critical Thinking, Mathematics, Social/Cultural 
Awareness, and Technological Awareness/ 
Computer Literacy. Each is responsible for as- 
sessing at least one general education outcome. 
This includes 

1. translating the general education outcome 
into specific and measurable student be- 
haviors, 

2. designing a rubric to score the student be- 
haviors, and 

3. setting target levels for success. 

As of the writing of this article, the Communi- 
cations and Mathematics subcominittees have 
completed these three steps, posted their work on 
the web for comment, and asked faculty to submit 



student artifacts that demonstrate the stated out- 
comes. Each of the other subcommittees are 
working to complete these steps for each of the 
remaining general education outcomes and should 
have them ready for comments during the fall se- 
mester. At that time, a call will be issued for arti- 
facts that demonstrate those outcomes. 

Using the rubrics for each outcome, the sub- 
committees will “grade” the artifacts submitted by 
instructors, and the aggregated results will be 
measured against the standards the subcommittees 
have established. The analysis of artifacts gath- 
ered in the summer and Fall will provide an op- 
portunity for the subcommittees to refine the ru- 
brics or standards (where needed) in preparation 
for a complete assessment of general education 
outcomes in the Spring of 2002. 

The results of the assessment will be reported 
to the College community. The results are ex- 
pected to generate "conversations" among faculty 
which will lead to concrete improvement actions. 
Implementing improvements will complete the 
first cycle of general education outcomes assess- 
ment. 

I invite you to visit the General Education Re- 
view Committee web site (see p.7 for address) for 
more information on our process, subcommittee 
membership, and contact information. I also hope 
you will review the outcome statements, rubrics, 
standards and possible assignments for literacy 
skills and understanding numerical data in prepa- 
ration for participating in ACC's fall and spring 
general education assessment activities. 

— Rex Peebles, Dean 
Social and Behavioral Sciences 
General Education Review Committee Chair 

peebles@austin.cc.tx.us 
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GeneraB Education Literacy SkiBBs: 

Outcomes, Rubrics, Standards, and Possibie Assignments 



General Education Outcome 1. “Possess suffi- 
cient literacy skills of writing, reading, speak- 
ing, and listening to communicate effectively 
above the 12*'' grade level.” 

Overview: In setting up the outcomes state- 

ments and outcomes rubrics, the Communications 
Subcommittee started with the Johnson County 
Community College model for General Education 
Review. However, we have made some changes 
that are significant. We decided not to include lis- 
tening as a communication skill to be evaluated be- 
cause of the inherent difficulty in finding appropri- 
ate artifacts and because listening is not specifi- 
cally identified in the SACS criteria. In addition, 
we decided to present the criteria for measuring the 
outcomes (rubrics) as a single set of criteria defin- 
ing minimum competence rather than as a range of 
criteria from superior to unsatisfactory. Again, our 
decision was based on the SACS criteria that an 
"institution must demonstrate that its graduates of 
degree programs are competent. ..." 

Outcomes Statements: Upon receipt of an as- 
sociate degree from Austin Community College, 
students should be able to: 

1. Read a document and demonstrate an un- 
derstanding of its content. 

2. Write a clear, well-organized paper using 
documentation where appropriate. 

3. Make a clear, well-organized verbal presen- 
tation. 

Outcomes Rubrics: Reading The following 
criteria will be used to measure the reading out- 
comes. Excluding literary works, which require a 
different methodology for interpretation, students 
are able to 

1. Summarize accurately the content of col- 
lege level reading material. 

2. Recognize main ideas that are stated or im- 
plied in the material and show how they are 
related to the other ideas and supporting de- 
tails in the material. i. 



3. Explain any arguments used and recog- 
nize the logic of those arguments. 

Outcomes Rubrics: Writing The follow- 
ing criteria will be used to measure the writing 
outcomes. Students are able to write an essay/ 
paper that is competent, responsive to the as- 
signment, and adequate. The essay/paper would 
include these typical features: 

1 . A thesis statement is apparent. 

2. An introduction and a conclusion are 
present. 

3. Most paragraphs are clearly organized, 
unified, and coherently presented with 
some transitions. 

4. Most of the sentences are clear; that is, 
they are understandable and punctuated 
appropriately. 

5. If research material is included, it is ade- 
quate to the assignment and documented 
appropriately. 

Outcomes Rubrics: Speaking To be 

judged as competent in speaking skills, students 
must meet the following minimum standards 
when making an oral presentation: 

1. The presentation has a clearly identifi- 
able design, complete with an introduc- 
tion and conclusion. The main points are 
easily recognizable and reasonably de- 
veloped. 

2. The presentation satisfies the minimum 
specific requirements of the assignment, 
such as number of references, time limit, 
or use of visual aids. 

3. The presentation is the student’s own 
original work. 

4. The topic is sufficiently focused with a 
clear sense of purpose (e.g. to inform, to 
persuade, to entertain.) 

5. Accurate citation of required number of 
sources. 

(Continued on page 4) 
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6. Eye contact, vocal control (rate and vol- 
ume), posture, body language, and de- 
meanor are satisfactory. The speaker has 
only occasional distracting mannerisms in 
posture, gesture, body movement, facial 
expression, or vocalizations (e.g. "um"). 

7. The speaker uses language correctly and 
does not detract from the message through 
gross errors of grammar, pronunciation, or 
articulation. 

Outcomes Standards: At least 75% percent 
of our students will demonstrate minimal compe- 
tence on the outcomes. Of course, we expect that 
a higher percentage of our students will demon- 
strate minimal competence. However, we have 
little to go on in setting the standard at the begin- 
ning of this process, so 75% has been agreed 
upon as a reasonable place to start. Notice that we 
are not surveying graduates, but all students, so it 
is reasonable that the standard is somewhat lower 
than it would be for graduates. 

Possible Assignments Reading: Students 
could present oral or written summaries and 
analyses of written messages which might include 



articles, text chapters, directions, and narratives so 
that the accuracy of the information included in the 
summary/analysis could be compared with the 
original. 

Possible Assignments Writing: Students 

could write an interpretative/analytical, informa- 
tive, or persuasive paper that would address a topic 
by explaining relevant ideas or by taking a position 
on an issue and defending it with a logical argu- 
ment. The paper could include sources, appropri- 
ately documented, where necessary. 

Possible Assignments Speaking: Students 
could make an oral presentation on a topic, either 
assigned or selected by student choice, and give an 
informative or persuasive presentation. The pres- 
entation might include outside sources where ap- 
propriate. The assignment could be audio- or 
video-taped for evaluation. 

— tennis Polnac, Professor 

English 

Communications Subcommittee Chair 

lpolnac@austin.cc.tx.us 



General Education Numerical Data Skills: 



Outcomes, Rubrics, Standards, 

Outcomes Statements; 

Upon receipt of an associate degree from Austin 
Community College, a student should be able to: 

1. Identify relevant data (numerical information in 
mathematical or other contexts) by 

a. extracting appropriate data from a problem 
containing extraneous data and/or 

b. identifying appropriate data in a word prob- 
lem. 

2. Select or develop models (organized representa- 
tion of numerical information, e.g., equation, table, 

O 




Possible Assignments 

graph) appropriate to the problem that represent 
the data by 

a. arranging the data into a table or spreadsheet 
and/or 

b. creating pictorial representations (bar graphs, 
or pie charts, or rectangular coordinate graphs, 
etc.) with or without technological assistance 
and/or 

c. selecting or setting up an equation or for- 
mula. 

74 (Continued on page 5) 
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3. Obtain and describe results by 

a. obtaining correct mathematical results, with 
or without technological assistance, and 

b. ascribing correct units and measures to re- 
sults which may or may not include writing an 
appropriate sentence interpreting the result. 

4. Draw inferences from data by 

a. describing a trend indicated in a chart or 
graph, and making predictions based on that 
trend and/or 

b. describing the important features of data pre- 
sented in a table or spreadsheet, and making 
predictions based on that trend and/or 

c. describing the important features of an equa- 
tion or formula, and making predictions based 
on those features and/or 

d. making reasonable estimates when given 
problems involving quantities in an organized 
or disorganized form and/or 

e. drawing qualitative conclusions about the 
original situation based on the quantitative re- 
sults that were obtained. 

Outcomes Rubrics: 

The following rubric will measure the mathe- 
matics outcomes. 

5 = All four major outcomes are demonstrated by 
the use of more than one subpoint per major 
outcome. 

4 = All four major outcomes are demonstrated. 

3 = Three major outcomes are demonstrated. 

2 = Two major outcomes are demonstrated. 

1 = Only one major outcome is demonstrated. 

0 = No major outcomes are demonstrated. 

The mathematics outcomes consist of four ma- 
jor outcomes, numbered 1 to 4. These major out- 
comes are each subdivided into several subpoints 



labeled by letters. A major outcome is demon- 
strated when at least one subpoint has been demon- 
strated, except for major outcome 3, where both 
subpoints must be demonstrated. In major outcome 
3, subpoint 3a will be demonstrated by at least 
70% accuracy of the items examined and subpoint 
3b will be demonstrated by at least 2 instances in- 
volving different measures. 

Outcomes Standards: 

At least 75 % of all ACC students surveyed 
will obtain a score of 3 or more on the mathemat- 
ics outcomes rubric. Obviously we want more stu- 
dents than this to meet what we consider to be 
minimum standards. But this seems like a reason- 
able place to start. We also think it would be a 
good idea to set a standard for the percentage of 4's 
we will obtain, in order to avoid the temptation to 
teach to minimum standards. Again, however, we 
don't feel ready to set a level for that standard yet. 

Possible assignments; 

The assignments which would provide the re- 
quired information could be all or a portion of a 
homework assignment, project, test, or other stu- 
dent work. Students should have the opportunity to 
provide at least four numerical results (since 70% 
accuracy is required). At least two of the answers 
should require different types of units of measure- 
ment. Types of problems chosen should include a 
word problem, and/or a problem containing extra- 
neous data. Students are expected to show how 
they organized their work (by means of formulas, 
equation, graphs, and/or tables) in order to arrive at 
their answers. They may be prompted to do this in 
the assignment, but the student must supply the de- 
tails. Also, students should be required to make at 
least one qualitative conclusion based on their 
quantitative work. 

— Mary Parker, Professor 

Math 

Mathematics Subcommittee Chair 

mparker@austin.cc.tx.us 
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Program Review and Unit-Level Effectiveness Assessment: 
Heating, Air Conditioning, and Refrigeration Technology (HART) 



I wish to relate my experience during the In- 
structional Program Review (IPR) conducted re- 
cently. The time invested in completing the 
process provided many rewards. 

After meeting for orientation, a self-study 
committee was assembled. We participated in a 
SWOT (strengths, weaknesses, opportunities and 
threats) analysis of our program which produced 
a vast array of ideas. 

The list of strengths, weaknesses, opportuni- 
ties and threats was prioritized and used to de- 
velop recommendations and action plans. This 
provided direction for the Heating, Air Condi- 
tioning & Refrigeration Technology (HART) 
Program for the next 2-5 years. 

The analysis of data required by the IPR was 
relatively straightforward, and the Office of Insti- 
tutional Effectiveness provided the data we re- 
quested. 

We decided to create three measures of effec- 
tiveness for the HART Program to evaluate next 
year. Our assessment plans include measures of 
student satisfaction, student success, and faculty 
rating. We developed several survey documents 
and plans for their implementation, as well as the 
methods for collection, analysis and reporting of 
the data. 

Unit Purpose 

The HART Department exists to graduate 
professionally trained people to the "Industry." 
Students completing our program are provided 
entry-level knowledge and skills. HART Certifi- 
cate Program graduates may expect to gain em- 
ployment in the Residential Heating and Air 
Conditioning Industry. HART AAS Degree Pro- 
gram graduates may expect to gain employment 
in the Residential and Light Commercial Heating 
and Air Conditioning Industry or the Commercial 
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Refiigeration Industry. Austin Community Col- 
lege maintains the HART program to provide a 
skilled resource of technicians to fill the many va- 
cant positions in the Heating, Air Conditioning 
and Refrigeration Industry. The major task of this 
program is to prepare learners with employer 
identified knowledge and skills. Students may be 
searching for a profession, may want to enter a 
new job market, or may desire formal training that 
will assist thern with being promoted from their 
present position. We also provide continuing edu- 
cation to those already employed in this industry 
that they may maintain or improve their job skills. 

Assessment Plan 

Outcome #: 1 

Outcome Title: Academic Goals. 

Intended Outcome: Students will achieve their 
intended academic goals, such as Level I Certifi- 
cate, AAS Degree, etc. 

Assessment Criteria: 70% of students will indi- 
cate on a final survey that they have achieved 
their intended academic goals from the HART 
program. 

Methodology: An initial survey of each student's 
intended academic goals will be conducted during 
their first HART course. A final survey, con- 
ducted at the time of the final exam in the stu- 
dents' last HART course, will be used for com- 
parison. HART faculty will conduct the initial and 
final surveys. HART staff will collect, sort and 
enter the responses into a database/spreadsheet. 
HART Program Coordinator will analyze the data 
after three consecutive semesters beginning Sum- 
mer Semester 2001 and will report the Summary 
and Analysis of Data (results, improvements, im- 
pact, etc.) by the end of the academic year, to the 
Dean of Applied Technology, HART Advisory 
Committee, HART faculty and staff, and the Unit- 
Level Effectiveness Assessment Database. 

(Continued on page 7) 
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(Continued from page 6) 

Outcome #: 2 

Outcome Title: Professional Goals 

Intended Outcome: Graduates that intend to, will 

find employment in the HART Industry. 



Data (results, improvements, impact, etc.) hy the 
end of the academic year, to the Dean of Applied 
^Technology, HART Advisory Committee, HART 
faculty and staff, and the Unit-Level Effective- 
ness Assessment Database. 



Assessment Criteria: 70% of all HART gradu- 
ates that intended to will, within one year after 
completing their academic goals, he employed in 
the HART Industry. 

Methodology: An initial survey, of HART stu- 
dents’ intended professional goals, will he con- 
ducted hy HART faculty. HART staff will collect, 
sort and enter responses into a database/ 
spreadsheet. HART graduates will be contacted 
by mail with a Graduate Employment Survey. 
Graduates will have been graduated one year from 
the Spring semester. HART staff will mail the 
Graduate Employment Survey during the summer 
semester. Responses will be collected, and data 
will be entered into the database/spreadsheet by 
HART staff. The HART Program Coordinator 
will report the Summary and Analysis of Data 
(results, improvements, impact, etc.) by the end of 
the academic year to the Dean of Applied Tech- 
nology, HART Advisory Committee, HART fac- 
ulty and staff, and the Unit-Level Effectiveness 
Assessment Database. 

Outcome #: 3 

Outcome Title: Instructional Quality 
Intended Outcome: HART Students will rate the 
instructional quality of the HART Program as 
“very good” or “excellent.” 

Assessment Criteria: The average of the com- 
bined HART faculty rating will be “very good” or 
“excellent.” 

Methodology: The annual Faculty Rating Report 
will be used as the source of data. The HART Pro- 
gram Coordinator receives the report and will av- 
erage the ratings as follows: Excellent = 5; Very 
Good = 4; Good = 3; Needs Improvement = 2; 
Unsatisfactory = 1 . The HART Program Coordi- 
nator will report the Summary and Analysis of 

Effectiveness jReuzew fall 2001 




I found the entire instructional review process 
enlightening. Although the effort was challeng- 
ing, and the rewards yet to be realized, we expect 
to be able to effectively evaluate our programs’ 
success and implement changes where warranted. 

— Dan Foust, Task Force Chair 
Heating, Air Conditioning, 
and Refrigeration Technology 

danfoust@austin.cc.tx.us 



Related Addresses 

Assessment of General Education Outcomes, 

General Education Review Committee Website: 
www2.austin.cc.tx. us/gened 

Internal Customer Satisfaction: 

www2.austin.cc.tx.us/oiepub/publications.htm#Studies 

ACC Self-Study Website: 

www2 . a u st i n . cc. tx . u s/sa cs 

Southern Association of Colleges and Schools 

(SACS) home page: 

sacscoc.org/index.asp 

SACS Criteria for Accreditation: 
sacscoc.org/criteria.asp 

Unit-Level Effectiveness Assessment Database 

(U-LEAD): 

www2.austin.cc.tx.us/oiepub/unitlevel.htm 
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Internal Customer Satisfaction Survey Results 



In order to provide the College with infor- 
mation to enhance its ability to meet the needs 
of its employees, an Internal Customer Survey 
of Fall 2000 services was administered in 
Spring 2001. The results of this survey were 
analyzed to determine which areas of the Col- 
lege were best meeting employee expectations 
and which were per- 
ceived as being less 
responsive. Areas 
with the lowest over- 
all ratings were pro- 
vided with more de- 
tailed analysis and 
will develop plans to 
address the concerns 
expressed by ACC 
employees. The fol- 
lowing report focuses 
on college-wide ser- 
vices. 

The Internal Cus- 
tomer Survey form 
was based on a 
scanable form that had 
been developed and 
administered in 1996. 

It was modified in Fall 

1999 and again in Fall 

2000 to reflect the 
then-current organiza- 
tion of the college and 
was reviewed and ap- 
proved by the Executive Vice Presidents and 
the Employee Association Presidents. 

In Fall 2000, two survey forms were cre- 
ated, one covering College-wide services and 
the other covering Campus-specific services. 
The College-wide services included 80 offices 
organized by administrative area during Fall 
2000. To assist respondents in identifying the 
services each office provides, a list of offices 
with a one-sentence description provided by 
the EVPs was included with each survey form. 
This was in response to concerns expressed on 
the previous survey indicating that many em- 



ployees do not recognize the specific name of an 
office. For each office, respondents were given 
an opportunity to indicate if they had requested 
or received services during the past year. They 
then rated the Promptness, Attitude and Overall 
Service of the offices they had used. 

In February 2001, the 
questionnaire was mailed to 
2,206 ACC employees. It 
was mailed to home ad- 
dresses to ensure that those 
without campus mailboxes 
would receive it. The em- 
ployee address lists were pro- 
vided by Human Resources 
and included all Full-time 
and Adjunct Faculty, as well 
as all Profession/Technical, 
Classified, and Administra- 
tive employees. Because of 
their limited interaction with 
the College and minimal re- 
sponse rate in 2000, hourly 
employees were not included. 
The returned forms were 
scanned into a data file and 
the data were aggregated by 
office for each question. In 
the following analysis, results 
for the College-wide services 
are based on responses of 
only those respondents who 
indicated that they had re- 
quested or received the services of a specific of- 
fice during the past year. 

Return Rate 

The overall return rate for the College-wide 
form was 22% or 493 employees, up from 15% 
in 2000. Rates varied greatly by employee 
group, but were similar in both years. 

The highest return rate was 55% of the 38 
administrators. The number and percent of Full- 
time Faculty returning the questionnaire in- 
creased from 26% to 29%, and the rate of Pro- 
fessional/Technical increased from 25% to 27%. 



In general, compared to last year: 

• The overall response rate increased from 
about 15% to about 22%. 

• The most highly used areas continued to 
be in Human Resources, but there were 
differences by employee group. 

•For Faculty, the most highly used offices 
included Faculty Development and Faculty 
Evaluation. 

• The overall level of satisfaction of respon- 
dents improved. The percent of Satisfied 
responses to Overall service increased 
from 58% to 61%. 

•Increases in the level of satisfaction were 
most notable in offices related to Market- 
ing. 

•Offices experiencing high levels of dissat- 
isfaction continued to be in the Human Re- 
source area, especially in terms of prompt- 
ness. 
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The response rates for both the Classified Staff 
and Adjunct Faculty declined slightly, from 20% to 
19% for Classified Staff and 13% to 12% for Ad- 
junct Faculty. 



Response Rates by Employee Group 


Employee Group 


2001 Survey 
Response Rate* 


2000 Survey 
Response Rate* 


Full time faculty 


29% 


26% 


Adjunct faculty 


12% 


13% 


Professional/ Technical 


27% 


25% 


Classified 


18% 


20% 


Administration 


55% 


43% 


Total 


22% 


15% 


*The percent of each group that returned the form. 



The largest percentage of all respondents were 
Adjunct faculty, while respondents who work in ad- 
ministration comprised the smallest percentage of all 
respondents. 

Percent of Total Respondents by Employee Group 



Employee Group 


% of All 
2001 Survey 
Respondents 


% of All 
2000 Survey 
Respondents 


Full time faculty 


23% 


22% 


Adjunct faculty 


24% 


32% 


Professional/Technical 


18% 


16% 


Classified 


17% 


20% 


Administration 


4% 


3% 


Not reported/ Other 


13% 


7% 


Total 


100% 


100% 



Use of Services 

The number of respondents requesting or receiv- 
ing services from each office ranged from a high of 
334 (up from 280 in 2000) for the Payroll Office, to 
a low of 25 for the Community Programs Office. 

High Use offices, those having 25% or more of 
the total number of respondents indicting they had 
used the service in the past year, included Human 
Resources, Faculty Development and Evaluation, 
the Computer Help Desk, and Mail Services. 

Overall Service 

The areas with the highest percentage of respon- 
dents satisfied with overall service — about 90% of 
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respondents — included Duplication Services at 
HBC, Library Services, and the Health Profes- 
sions Institute. Other offices with high rates of 
satisfaction with overall service included the In- 
terpreter Services, the Office of the Dean of 
Health Sciences, and Video Services. 

The general level of satisfaction appears to 
have increased between 2000 and 2001. In 
2000, 58% of all responses for Overall Service 
were either Satisfied or Very Satisfied. In 2001, 
this increased to 61%. At the same time. Neu- 
tral responses remained steady at 19% and Dis- 
satisfied/Very Dissatisfied responses decreased 
from 23% to 20%. 

Several individual offices experienced a 
high degree of change in satisfaction with over- 
all service ratings they received. Of the 10 of- 
fices with the greatest increase, three had a 
change of 20% or more; the Publications Of- 
fice, Student Recruitment and Adult Basic Edu- 
cation/ESL. Only the TASP office registered a 
decline in satisfaction with overall service of 
over 20%. None of these offices, however, 
were High Use areas. 

High Use areas with increases in satisfaction 
with overall services of 10% or more included 
the Computer Help Desk, Marketing and Public 
Information, and Open Campus (Distance 
Learning). High Use areas with decreases in 
satisfaction with overall services of 10% or 
more included HBC Room Scheduling and In- 
ventory/Receiving. The table on the following 
page displays these results. 

Use and Dissatisfaction 

There were thirty “high use” offices. Of 
these, 14 are of particular interest because over 
20% of respondents indicated dissatisfaction 
with the services provided during the past year. 

Of the 14 offices, six were in the Human 
Resources area. Three of the High Use/High 
Dissatisfaction offices provide services mainly 
to faculty: Faculty Evaluation and the Deans of 
Arts and Humanities and Math and Sciences. 
All but Faculty Evaluation, HBC Room Sched- 
uling and Continuing Education were also High 
Use/High Dissatisfaction offices in 2000. 

(Continued on page 10) 
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Satisfaction with Overall Services by Percent Change 


Highest Increase in Satisfaction 






Highest Decrease in Satisfaction 




Office/Service 


2001 Survey 


2000 Survey 


% 




Office/Service 


2001 Survey 


2000 Survey 


% 




%Satisfied 


%Satisfied 


Change 






%Satisfied 


%Satisfied 


Change 


Publications Office 


63% 


41% 


22% 




Dean, Communications 


70% 


77% 


-7% 


Student 


62% 


41% 


21% 




AVP for Academic 


69% 


76% 


-8% 


Recruitment 










Affairs 








Adult Basic 


74% 


54% 


20% 




Dean, Computer Studies 


74% 


81% 


-8% 


Education/ ESL 










& Advanced Technology 








Press Relations 


53% 


41% 


12% 




EVP for Instructional 


54% 


62% 


-8% 


Computer Help 


64% 


52% 


11% 




Affairs 








Desk 










Dean, 


48% 


57% 


-8% 


IT Programming 


58% 


47% 


11% 




Continuing Education 








Marketing & Public 


53% 


42% 


11% 




Accounts Payable 


46% 


55% 


-9% 


Information 










Risk Management 


76% 


85% 


-9% 


International Edu- 
cation Programs 


64% 


53% 


10% 




Inventory/Receiving 


62% 


72% 


-11% 


Open Campus 


78% 


68% 


10% 




HBC Room Scheduling 


53% 


64% 


-11% 


ACCNet Services 


63% 


54% 


9% 




TASP Office 


40% 


62% 


-23% 



Satisfaction of Employee Groups 

For purposes of this analysis, employees 
were grouped into three categories: Full-time 
Faculty, Adjunct Faculty, and Non-faculty. 
Rates of dissatisfaction varied by employee 
group. For example, while all groups were dis- 
satisfied with the Payroll Office, the percentage 
dissatisfied ranged from 65% of Full-time Fac- 
ulty respondents to 29% of Non-faculty, with 
Adjunct Faculty being in the middle at 36%. 

Full-time faculty expressed the highest 
levels of dissatisfaction in general. In 17 of the 
30 High Use offices, their ratings of dissatisfac- 
tion were 5% above the average for all respon- 
dents. This group was most dissatisfied (over 
60%) with Employment/Hiring Services, Pay- 
roll Services, Compensation/Classification, and 
the office of the AVP for Human Resources. 

Adjunct faculty generally had lower lev- 
els of use than other groups. Their highest levels 
of dissatisfaction were with the Telephone Sys- 
tem (60%), Employment/Hiring Services (58%), 
and the Office of the AVP for Human Re- 
sources (63%). 
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Non-faculty tended to be less dissatisfied 
overall than faculty groups. Their highest levels of 
dissatisfaction occurred with Employment/Hiring 
Services (50%), Faculty Evaluation (39%) and Per- 
sonnel Records (31%). 

Satisfaction with Promptness and Attitude 

The ratings discussed thus far have been based 
only on the responses to Overall Service. To assist 
in developing an additional set of information, two 
other dimensions of quality service were also rated. 
Promptness, and Service Attitude. Trends in these 
dimensions may assist areas in developing plans to 
improve their effectiveness in meeting employee 
needs. 

The table on the opposite page shows the 
percentage of respondents indicating dissatisfaction 
with Promptness was higher than that of Overall 
Service for 1 1 of the 30, or 37%, highly used of- 
fices. However, compared with the results of the 
2000 survey, this is a decline of 20%. This sug- 
gests that, in general, offices are increasingly 
prompt when providing services to others. 

8 Cl (Continued on page 11) 
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Promptness continues to be an issue for 
several offices that affect many employees, includ- 
ing: Employment/Hiring Services, Compensation/ 
Classification, Accounts Payable, Employee Bene- 
fits, and Purchasing. 



On the other hand. Service Attitude ratings 
tended to track with Overall Service ratings; most 
Service Attitude percentages were within 5 per- 
centage points of the Overall Service ratings. 

— Martha Oburn, AVP 
Institutional Effectiveness 

mlo@austin.cc.tx.us 



Level of Dissatisfaction: Comparison of Dimensions of Quality Service 
of Highly-Used Offices/Services Listed in Order of Use 



College-wide Offices/Services 


Level of Dissatisfaction 




Overall service 


Promptness 


Attitude 


Payroll Office 


41% 


42% 


38% 


Employee Benefits 


25% 


30% 


27% 


Computer Help Desk 


20% 


13% 


17% 


Faculty Development 


18% 


23% 


13% 


Mail Services 


8% 


10% 


7% 


Faculty Evaluation 


42% 


39% 


30% 


Employment/Hiring Services 


44% 


50% 


42% 


Library Services 


5% 


4% 


5% 


Staff Development 


19% 


18% 


20% 


Campus Police (HBC) 


9% 


9% 


11% 


Admissions & Records 


15% 


15% 


13% 


Personnel Records 


37% 


37% 


33% 


Cashier 


18% 


21% 


21% 


Duplication Services (HBC) 


4% 


5% 


3% 


Purchasing 


19% 


29% 


20% 


Telephone system 


15% 


15% 


16% 


Instructional Technology 


11% 


11% 


10% 


ACCNet Services 


22% 


30% 


23% 


Compensation/Classification 


39% 


44% 


35% 


Accounts Payable 


26% 


33% 


24% 


AVP for Human Resources 


38% 


40% 


37% 


Maintenance (Repairs) 


15% 


23% 


14% 


Dean, Social & Behavioral Sciences 


9% 


12% 


10% 


HBC Room Scheduling 


25% 


27% 


25% 


Dean, Math & Science 


21% 


28% 


24% 


Open Campus (Distance Learning) 


9% 


15% 


10% 


Continuing Education 


21% 


26% 


23% 


Executive Dean NRG/RGC 


18% 


15% 


23% 


Marketing & Public Information 


21% 


21% 


24% 


Dean, Arts & Humanities 


25% 


27% 


25% 
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SACS Self-Study: Assessing the Present and Planning the Future 

Austin Community College (ACC) is a member of the Commission on Colleges of the Southern Association of Col- 
leges and Schools (SACS). This is the recognized accrediting agency for post-secondary institutions in the eleven southern 
states (Alabama, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, Texas and 
Virginia) and in Latin America. The Commission accredits institutions that award associate, baccalaureate, master's, spe- 
cialist's or doctoral degrees. 

Periodically, currently once every ten years, each member institution of SACS undertakes a self-study that involves 
faculty, administrative personnel, professional, technical and classified staff, students, and trustees in a close exarnination of 
the institution. At the culmination of the self-study, the Commission on Colleges sends a committee of peers and consultants 
to evaluate the institution's effectiveness in reaching its stated goals and in complying with the Criteria for Accreditation, and 
to consult with the institution on issues important to its advancement. The visiting committee develops a consensus of its find- 
ings and completes a formal written report. The written report of the visiting committee provides the basis on which the Com- 
mission will decide on reaffirmation of accreditation for ACC. 



Reaffirmation of SACS Accreditation 

Austin Community College is beginning its 
journey toward reaffirmation of accreditation in 
2003. The SACS Committee of Peers will be vis- 
iting ACC March 3-6, 2003. We look forward to 
this opportunity to evaluate our performance as a 
college, and to ascertain our strengths as well as 
areas which need improvement. Our goal is to en- 
sure a strong future for our students and our insti- 
tution. Following the framework outlined for the 
alternate self-study model, the College will par- 
ticipate in the following compliance and strategic 
analyses. 

Assessing the Present 

A select group of faculty, administrators, and 
other staff will conduct a comprehensive compli- 
ance audit to assess all aspects of the College's op- 
erations in order to demonstrate compliance with 
the "must" statements in the Criteria for Accredi- 
tation. We will develop recommendations and a 
corresponding action plan to ensure that the Col- 
lege meets all standards by Spring 2003. 

The specific goals of the compliance audit are to: 

• Sharpen the focus of the institution's stated 
purpose and mission and to clarify its 
goals 

• Provide an opportunity to identify the criti- 
cal issues and concerns facing the institu- 
tion and generate new or improved ap- 
proaches to these issues 



• Increase efficiency and productivity of the 
operational units of the institution 

• Strengthen institutional research 

Faculty and staff who are not currently serving 
on a compliance committee may be asked to assist 
in gathering data, supply an administrative office, 
for example Human Resources, with documents, 
fill out and return surveys or participate in focus 
groups or other forms of research. Additionally, 
all faculty and staff need to read the ACC SACS 
electronic or printed newsletter and regularly check 
the ACC SACS Accreditation Visit website to stay 
informed on the process and progress of the self- 
study and provide input to the process. 

Planning the Future 

A separate select group of volunteers, inter- 
ested faculty, administrators, and other staff widely 
representative of the entire campus community, 
will concentrate on the focused strategic self-study. 
They will conduct an in-depth examination of four 
critical areas for enquiry believed to affect student 
learning and success in the future. A fifth commit- 
tee will focus on an action plan based on the find- 
ings of the other four committees. 

The goals of the strategic focus are to: 

• Analyze the skills and learning attributes of 
the college's prospective students 

• Identify what students want from their col- 
lege experiences at ACC 

^ (Continued on page 1 3) 
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• Determine what students expect the learn- 
ing environment to be 

• Determine appropriate integration of tech- 
nologies throughout the institution 

• Determine what is necessary to prepare the 
College's faculty to use new technology 
for meeting students' needs 

• Identify appropriate teaching and learning 
resources and determine how to provide 
them 

• Implement findings and recommendations 
within the fabric of the College through an 
on-going mechanism to update the envi- 
ronmental scans and the emerging new 
technologies and apply recommendations 
through the ongoing three-year Compre- 
hensive Master Plan. 

The rationale for this focus is, in a word, Change. 
Change is everywhere in the higher education lit- 
erature these days. Prognosticators assert that we 
are now in the business of creating "life-long 
learners." That means we must be producing citi- 
zens who have creative problem-solving skills, 
critical thinking skills, communication and learn- 
ing skills, and perhaps most importantly, an abil- 
ity to cope with change in the economic, social 
and workplace contexts. 

The SACS Alternate Self-Study process offers 
us an opportunity to meet this challenge. How 
will we prepare our students to be competent con- 
sumers of information? What role will informa- 
tion technology play in teaching? Will we be able 
to compete with on-line education from electronic 
providers? How will we respond to the increasing 
gap between financial demands and scarce re- 
sources? Will we remain "student-centered" or 
move toward a student as client model? How will 
we ensure we are flexible enough to respond to 
the changing needs of our community? What 
steps will we take to prepare students for the 
workplace? Do we want faculty to share in gov- 
ernance of the college? On and on the questions 

go. 



Self-Study Strategic Focus 

Change and growth have been the hallmark of 
Austin Community College since it opened its 
doors in September 1973. The need for ACC to 

O 
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prepare students for an increasingly technical 
world characterized by constant change has influ- 
enced our choice of a strategic focus for our re- 
accreditation efforts. Lifelong learning, critical 
thinking skills, and information literacy skills are 
more and more important for our students and 
ourselves. So we came up with a very fancy title 
for our strategic focus to reflect our wish to better 
understand our students' needs and to develop 
strategies to meet those needs for the next ten 
years: Infusing 21st Century Innovation into 

Learning: A Student-Centered Examination of 

New Technologies, Faculty Roles, and Institu- 
tional Structures. 

With so many things changing so quickly and 
with new irmovations around every comer, how 
can we predict what we will need to be doing two 
years from now, much less ten years? This has 
been the first hurdle for the Self-study strategic 
focus committees to clear . With a little help from 
futurist. Dr. Peter Bishop, professor at the Univer- 
sity of Houston, Clear Lake, those of us who vis- 
ited with him recently may have a little better idea 
of how to "Start Here" and "Get There." 

We all know that change is inevitable, and, 
even if we don't all like it, chaos (at least some 
chaos) is both good and necessary to effect real 
change. Even so, there will be constants; for in- 
stance, our values and aspirations for teaching and 
learning outcomes generally will not change. Dr. 
Bishop helped us focus on the fact that we may 
not be able to predict what will happen, but we 
can predict what could happen. And, we can also 
help keep what we don't want to happen from sud- 
denly surprising us and knocking us off course 
unnecessarily. 

One way of achieving our goal is to say "What 
do we want to be happening for our students at 
ACC ten years from now?" Once we define this, 
then we can use a process called "back-casting" to 
say things like "Where do we need to be five 
years from now to get to where we want to be 10 
years from now?" We need a vision, and then we 
need to concentrate our efforts on forecasting 
what we will need to do and where we will need 

(Continued on page 14) 
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(Continued from page 13) 

to be at different stages of the time-line to achieve 
the vision. When we create this vision, we will 
include "will", "might" and "should" in that vi- 
sion. When we create that vision, we want the 
Austin Community College community to be able 
to articulate that vision in a very short statement. 
Most of all, we want that vision to be affirmed 
through the College's strategic planning process 
and to show up as meaningful, measurable steps 
in the rolling three-year master plan. 

Want to Participate? 

As stewards of the SACS self-study process, 
we want as much college input and participation 
in this strategic focus as we can get. This is par- 
ticularly a very important opportunity for faculty 
to have input into what their jobs and their class- 
rooms (virtual or physical) will look like ten years 
from now. 

Over the next two years, you will likely hear 
much more about our Alternate Self-Study, and 
will have many opportunities to participate. We 
hope you will. Our focus is purposefully broad 
and calls into question our institution's structures 
and processes, our faculty and staff development, 
our use of technology and innovations in teaching, 
and our understanding of our future students' 
needs. To become more familiar with the Alter- 
nate Self-Study, or to find ways to participate, 
contact us. We are looking forward to helping you 
get involved. 

— Cary Sowell, Librarian 
Steering Committee Director 

cary@austin.cc.tx.us 

— Lynn Beaman, Professor 
Human Services 
Compliance Committee Chair 

lbeaman@austin.cc.tx.us 

— Mark Butland, Associate Professor 

Speech 

Strategic Focus Committee Chair 

butland@austin.cc..tx.us 
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Closing the Loop 

What improvements are instructional units 
making as a result of assessing student learn- 
ing outcomes? The improvement action plans 
documented into the Unit-Level Effectiveness 
Assessment Database (U-LEAD) were exam- 
ined to answer this question. This article pre- 
sents a brief summary of the findings from 
that examination. 

Instructional units at ACC are using the 
results of their effectiveness assessments to 
improve many aspects of their programs. 
These include the following: 

• Curriculum or the tools used in providing 
instruction (software/labs/books, etc.) 

• Training or professional development for 
instructors 

• Relationships with external entities 
(employers, licensing agencies, task 
forces, etc.) 

• Relationships with students 

• Funding and instructional resources 

• Common syllabi, exams, grading policies, 
instructional objectives, and instructional 
methods 

• Revising the assessment tool or methodol- 
ogy 

• Re-assessing the same outcomes for con- 
firmation of positive results 

• Assessing different outcomes because cur- 
rent ones were successful 

• Continuing already successful practices 

• And “no improvements needed.” 

Unfortunately, some units have not docu- 
mented any improvement action plans into the 
database. It is critical to complete this step, as 
our accrediting agency will be looking for the 
improvements that have resulted from our ef- 
fectiveness assessment efforts. 

— Roslyn Wallace, Coordinator 
Institutional Assessment 

rwall@austin.cc.tx.us 
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Graduate Outcomes — Workforce Detail 



To determine whether graduates of public commu- 
nity colleges in Texas are employed, or continuing their 
education, or both, the Texas Higher Education Coordi- 
nating Board (THECB) uses state employment records 
and various THECB data to locate students following 
graduation. However, these data may overlook self- 
employed graduates and graduates who have transferred 
to private or out-of-state schools. Additionally, the em- 
ployment records do not indicate whether the graduate is 
employed in the field for which the degree or certificate 
was granted. 

Each year, ACC surveys its graduates of the previ- 
ous academic year to provide data that supplement the 
THECB ’s tracking data. 

The ACC 1999-2000 Graduate Survey was distrib- 
uted during the Spring of 2000 to 1103 students who 
graduated during the 1999-2000 academic year. A total 
of 173 (15.7%) questionnaires were returned. Not all re- 
spondents answered every question in the survey. 

Graduates were asked to respond to questions about: 
■employment status after graduation 
■whether their degree or certificate was required 
for their job 

■any further education beyond their ACC work 
■any volunteer work involving ACC training 
■their opinion of how well ACC prepared them 
for employment or further education 



The chart below illustrates that of the 173 graduates 
responding, 145 (83.8%) are, or were at one time af- 
ter graduating, employed. Of those, 122 graduates 
(70.5% of all respondents) were employed in a job 
that was related to their ACC training. Additionally, 
three respondents (1.7%) volunteered in an area re- 
lated to their ACC training even though they did not 
use their training in a job. 

Most (150 or 86.7%) of the graduates responding 
to the survey rated their ACC preparation for employ- 
ment or further education “excellent” (54.3%) or 
“satisfactory” (32.4%). 



Please rate how well you feel ACC prepared 
you for ennploynnent or further education. My 
preparation was... 




Respondents 
n % 


Excellent 


94 


54.3% 


Satisfactory 


56 


32.4% 


Good only in some areas 


16 


9.2% 


Fair 


3 


1.7% 


Inadequate 


2 


1.2% 


Did not respond 


2 


1.2% 


Total 


173 


100% 



Follow-Up Survey Results: 1999-2000 Graduates 
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The following questions pertain to documenting effec- 
tiveness assessment plans into the web-based Unit-Level 
Effectiveness Assessment Database (U-LEAD). 




Q: I know the deadline for documenting my 
unit’s effectiveness assessment activity into the 
U-LEAD is in June, but that is a very busy time 
for faculty. Is there anything we can do to re- 
lieve the pressure of that deadline? 

A: YES! If there is a better time to document 
your effectiveness assessment activities, please do 
it then. The only requirement is that by the dead- 
line, all the steps have been documented for the 
current academic year. For example, you might 
document the outcomes, criteria, and methodolo- 
gies you intend to implement for the 2001-2002 
academic year in the Fall of 2001 (perhaps after the 
first-of-th e-year task force or program advisory 
committee meeting). Then, after you have con- 
ducted the assessment — say as part of the final 
exam in the course being assessed — you could 
document the results and improvement action plans 
in the spring or summer. 

Additionally, the appropriateness of the June 
deadline is being discussed, and could change to a 
fall deadline. If that happens, you will be in- 
formed. 

Q: What if my unit does not have the results 
of the effectiveness assessment by the deadline? 

A: This sometimes happens when the method- 
ology of the assessment plan entails student per- 
formance that occurs over a longer time period 
than one academic year. In such cases it is impor- 



tant to document what is happening. In these 
cases, the unit would indicate “results are not 
currently available” and “improvement action 
will be determined when results are avail- 
able.” (Do not leave these fields blank.) Then, 
when the results have become available and the 
improvement plan created, go back and enter 
them in the U-LEAD. Of course, you will have 
to remember to do this in the next academic year 
or whenever the results are available. 

Q: How do I get to the U-LEAD? 

A: From the ACC Intranet homepage 
(http://accweb.austin.cc.tx.us) 

• click on the link to the Office of Institu- 
tional Effectiveness . 

• From there click on Unit-Level Institu- 
tional Effectiveness Database. 

• Then click on Unit-Level Effectiveness 
Assessment (ULEA) Database. 

• Click OK in the message box that ap- 
pears. You should be at the front page of 
the documentation database. 

Follow the instructions on this page to either 
create a new record or view/edit an existing re- 
cord. You will need a User ID and password to 
submit a new record. To obtain those, or if you 
have any problems accessing the database, or if 
you need other assistance, please contact Roslyn 
Wallace at 223-7585 or rwall@austin.cc.tx.us. 
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ACC Student Engagement.. .Educational Bliss? 



In Fall 2001, ACC received the results from its 
participation as a test institution in the summer pi- 
lot of the new Community College Survey of Stu- 
dent Engagement (CCSSE). This article presents an 
analysis of ACC student responses to the first item 
of the survey; a complete analysis will be published 
later this spring. 

Survey Design 

The CCSSE, modeled on the National Survey of 
Student Engagement, is a recent project of the 
Community College Leadership Program at The 
University of Texas at Austin. The CCSSE, as the 
NSSE does for students attending four year institu- 
tions, assesses the extent to which community col- 
lege students engage in educational practices asso- 
ciated with high levels of learning and develop- 
ment. The survey items focus on learning and learn- 
ers. 

A substantial body of research indicates a key 
factor for student learning is the quality of effort 
students devote to educationally purposeful activi- 
ties. The greater the effort and time expended by 
students in the opportunities afforded by their col- 
leges, the greater the likelihood of academic and 
personal growth, satisfaction with the institution, 
and persistence within the educational system. The 
CCSSE is designed to examine students’ participa- 



tion in activities identified as “good practices” 
that research has shown to be directly related to 
student retention and other positive student out- 
comes. The survey items address 

• How students spend their time, 

• What they feel they have gained from their 
classes, 

• How they assess the quality of their inter- 
actions with faculty, counselors and peers, 

• What kinds of work they are challenged to 
do, and 

• How the college supports their learning. 

Additionally, the survey gathers student edu- 
cational and socio-economic demographic data. 
A copy of the survey instrument may be viewed 
on the web at http.V/www.ccsse.org/survey. 

Results for Item One 

The pilot version of the CCSSE was admin- 
istered in class to a randomly selected group of 
community college classes in summer 2001. 
The survey sample included 29 ACC classes, a 
total of 458 ACC respondents. 

Item 1 consisted of a question, “In your ex- 
perience at this college during the current school 

(Continued on page 2) 
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year, about how often have you done each of the 
following,” followed by a list of experiences re- 
lated to student engagement. For purposes of this 
analysis, these were grouped into the following 
categories: 

Experiences related to 

• Classroom participation and preparation 

• Interactions with instructors 

• Assignments 

• Interactions outside class 

• Using technology 

• Effort or determination 

Students were asked to indicate the frequency 
of their participation in each of the experiences 
using the response options of “Never,” 



“Sometimes,” “Often,” and “Very Often.” 

Table 1 displays a summary of students’ re- 
sponses to Item I, organized by category. 

In terms of classroom participation and prepara- 
tion, most students participating in the survey ask 
questions or contribute to class discussions, but 
many may not be prepared for class. 

• Fewer than one in ten students (7%) never 
asked questions or contributed to class discus- 
sions (see Chart A, p. 3). 

• Almost the same percentage very often (8%) 
came to class without completing readings or 
assignments. 

In their interactions with instructors, students 



Table 1 : 2001 Community College Study of Student Engagement 
Responses to Item 1 



1) In your experience at this college during the current school 


Total 


Never 


Sometimes 


Often 


Very Often 


year, about how often have you done each of the following? 


# 


# 


% 


# 


% 


# 


% 


# 


% 


Classroom participation and preparation 


Asked questions in class or contributed to class discussions 


458 


30 


7% 


185 


40% 


143 


31% 


100 


22% 


Came to class without completing readings or assignments 


457 


69 


15% 


260 


57% 


91 


20 % 


37 


8% 


Interactions with Instructors 


Discussed grades or assignments with an instructor 


457 


50 


11% 


192 


42% 


144 


32% 


71 


16% 


Talked about career plans with an instructor or advisor 


457 


214 


47% 


159 


35 % 


69 


15% 


15 


3% 


Discussed ideas from your readings/classes with instructors outside of 
class 


457 


260 


57 % 


141 


31% 


44 


10% 


12 


3% 


Assij 


gnments 


Worked on a paper or project that required integrating ideas or informa- 
tion from various sources 


457 


73 


16% 


145 


32% 


163 


36% 


76 


17% 


Worked with other students on projects during class 


458 


116 


25% 


179 


39% 


127 


28% 


36 


8% 


Made a class presentation 


458 


209 


46% 


168 


37% 


51 


11 % 


30 


7% 


Participated in a community-based project as a part of a regular course 


458 


410 


90% 


36 


8% 


10 


2% 


2 


0% 


Interactions with others outside of class 


Discussed ideas from your readings or classes with others outside of 
class (students, family members, coworkers, etc.) 


458 


63 


14% 


178 


39% 


151 


33% 


66 


14% 


Worked with classmates outside of class to prepare class assignments 


458 


204 


45 % 


167 


36% 


70 


15% 


17 


4% 


Tutored or taught other students (paid or voluntary) 


456 


314 


69% 


98 


21% 


35 


8% 


9 


2% 


Had serious conversations with students of a different race or ethnicity 
than your own 


458 


102 


22% 


162 


35 % 


111 


24% 


83 


18% 


Had serious conversations with students who differ from you in terms of 
their religious beliefs, political opinions, or personal values 


458 


112 


24% 


152 


33% 


119 


26% 


75 


16% 


Technology 


Used email to communicate with an instructor 


458 


136 


30% 


159 


35 % 


96 


21% 


67 


15% 


Used an electronic medium (list-serv, chat group, Internet, etc.) to dis- 
cuss or complete an assignment 


458 


176 


38% 


119 


26% 


100 


22% 


63 


14% 


Effort/Determination 


Worked harder than you thought you could to meet an instructor’s stan- 
dards or expectations 


458 


63 


14% 


215 


47% 


139 


30% 


41 


9% 


Prepared two or more drafts of a paper/assignment before turning it in 


457 


137 


30% 


139 


30% 


116 


25% 


65 


14% 
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Chart A 

Asked questions in class or contributed to class 
discussions 




Percent of respondents 



are more likely receiving and discussing feedback 
on their performance than they are discussing ideas 
from their readings or their career plans. 

• About one in every ten students (12%) never 
received prompt feedback (written or oral) from 
their instructors and never discussed grades or 
assignments with their instructor (11%). 

• Nearly half (47%) never talked about career 
plans with an instructor or advisor, and 

• Just over half (57%) never discussed ideas from 
their readings or classes with instructors outside 
of class. 

Data on experiences related to assignments re- 
vealed most students are required to integrate ideas 
and information from various sources, but they are 
not usually required to work with other students or 
in the community as part of their course work. 



• Sixteen percent never worked on assign- 
ments/projects that required them to inte- 
grate ideas or information from various 
sources (see Chart B). 

• One of every four students (25%) reported 
they never worked with other students on 
projects in class. 

• Nearly half (46%) never made a class pres- 
entation 

• And most (90%) never participated in a 
community-based project as part of a regular 
course. 

Outside of class, students tend to discuss 
ideas from their class with others, though they 
rarely work on assignments with, or tutor, other 
students. 

• Almost half (45%) never work with class- 
mates outside of class to prepare class as- 
signments (see Chart C). 

Chart C 

Worked with classmates outside of class to 
prepare class assignments 




Percent of respondents 



Chart B 

Worked on a paper or project that required 
integrating ideas or information from various 
sources 



very often 



c often 
0) 

3 

O’ 

£ sometimes 



never 

0 % 10 % 20 % 30 % 40 % 

Percent of respondents 



1 


17% I 






3^ 






















• Just over two-thirds (69%) never tutored 
or taught other students (paid or voluntary) 

Although many students report they use 
technology in their college work, 

• Almost one third (30%) have never used 
e-mail to communicate with an instructor, 
and 

• Just over a third (38%) have never used an 
electronic medium (list-serv, chat group, 
Internet, etc.) to discuss or complete an as- 
signment. 

(Continued on page 4) 
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And finally, although few students (14%) re- 
ported they never expend a great deal of effort to 
meet an instructor’s expectations (see Chart D), 
more than twice that percentage (30%) never re- 
vise their work before turning it in (see Chart E). 

Chart D 



Worked harder than you thought you could to 
meet an instructor’s standards or expectations 



very often 



g often 
a> 

3 

O* 

£ sometimes 



never 



0% 1 0% 20% 30% 40% 50% 

Percent of respondents 
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Chart E 



Prepared two or more drafts of a paper or 
assignment before turning it in 




Percent of respondents 



This analysis is presented as a “snapshot” of a 
particular group of students at a particular point 
in time; conclusions based on these data should 
be drawn with caution. However, the data do 
provide a context for a dialogue regarding the 
role the College plays in engaging students in 
their college education. 



In light of the research that indicates (and that 
most faculty know experientially) student engage- 
ment is a key factor in student learning and reten- 
tion, these data raise several questions. 

• How can faculty identify, during the first 
week or so of class, which students are not 
engaged in their educational experience? 

• How can faculty determine which students 
are at risk, based on their level of engage- 
ment? 

• What can faculty and other staff do to en- 
courage and support students to become more 
engaged in their educational process? 

• Are there specific teaching strategies that can 
improve students’ level of engagement? 

• Are there specific strategies that student sup- 
port services can implement to increase the 
likelihood of students becoming engaged? 

The analysis of other items of the CCSSE 
should provide additional information pertinent to 
this dialogue. 

• Are these responses equally typical of stu- 
dents in the long semester as of those taking 
summer school classes? 

• Are they equally typical of first semester stu- 
dents as of those who have completed several 
semesters? 

• What factors are common to students who 
are more engaged in their education? 

In Spring 2002, ACC students will participate 
in the second administration of the CCSSE. The 
results of this spring survey will provide data for 
comparisons to the results of this past summer, as 
well as to the results of other community colleges. 

— Roslyn Wallace 
Coordinator, Institutional Assessment 
Office of Institutional Effectiveness 

HBC 
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General Education Assessment: 

Where the Rubric Meets the Road 



As was reported in the Fall 2001 issue of the 
ACC Effectiveness Review, the General Educa- 
tion Review Committee has been working on de- 
fining measurable general education outcomes 
and creating rubrics to use in examining student 
work that should demonstrate those outcomes. 
This fall, the communication outcomes rubrics 
were “tested” to determine reliability (the rubric 
yields the same results when more than one per- 
son is applying it) and validity (use of the rubric 
yields approximately the result one would ex- 
pect). This article reports the results of that pilot 
test. 

In October, 200 1 , five faculty (all of whom do 
not teach English) on the Communications sub- 
committee of the General Education Review 
Committee sat down together to apply the com- 
munications rubrics (for reading, writing, and 
speaking) to a set of student artifacts. Using the 
rubrics, they examined 24 student writing artifacts 
(in this case, papers from English Composition I, 
English Composition II, and Sophomore literature 
courses), twelve student reading artifacts (book 
reports from a history class), and four samples of 



student speeches (videotaped from a speech 
course) to determine how well the students demon- 
strated the outcomes for each skill area. 

For the most part, the scores were consistent 
across graders with never more than two graders 
disagreeing with the score for any single artifact. 
This indicates the rubric is reliable. Additionally, 
in the area of writing, the number of successful ar- 
tifacts increases with the exposure to the content of 
the courses. This indicates the rubric does in fact 
measure the skills that students should be able to 
demonstrate as they progress in their college ex- 
periences. The table below displays the results of 
the scoring session. 

The Communications sub-committee feels con- 
fident that the rubric can be used to determine stu- 
dents’ learning in the communications skill areas 
of general education. 

— Linnis Polnac, Professor 
English 
NRG 



Results of Pilot Test of General Education Communications Skiiis Rubric 



Area 

(Number of 
samples) 


Number of Samples 


Number 
of Faculty 
Scoring the 
Samples 


Number of Samples with 
No Score Disagreement 


Number of Samples 
with Score Disagreement 


Met criteria 


Did not 
meet 
criteria 


One Grader 
Disagrees 


Two Graders 
Disagree 


Writing (24) 


19 


5 


5 


15 


6 


3 


Reading (12) 


10 


2 


5 


8 


3 


1 


Speech (4) 


4 


0 


5 


4 


0 


0 
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Transfer Outcomes: Start Here. ..Get There and Succeed 



Recent transfer reports from the University 
of Texas at Austin and Southwest Texas State Uni- 
versity confirm that Austin Community College 
students perform well after transferring. The re- 
ports also show that ACC is still the largest feeder 
for both these institutions. Following is a sum- 
mary of each school’s report. 

The University of Texas at Austin 

The University of Texas at Austin reports 
the following information regarding a cohort of 
ACC students who entered the University in fall 
2001. 

• ACC students make up nearly a quarter of 
the total transfer students population for 
UT. Of 990 total applications from ACC 
students, 491 (49.6%) were admitted and 
473 (96.3%) actually enrolled showing a 
very high “admit-to-show” rate. 

• The average GPA of those who enrolled 
was 3.46 and the average number of hours 
completed prior to transferring was 53, 
with the majority of students having at- 
tended only ACC prior to transferring to 
UT. 



Southwest Texas State University 

The report from Southwest Texas State 
University reflects performance and retention of 
ACC students during their first academic year (fall 
2000 and spring 2001) at SWT. 

• Overall, 655 students transferred to SWT 
during the academic year making up nearly 
a third of the total transfer population. 

• While the decision GPA' for ACC students 
(2.76) was slightly lower than the average 
for all two-year college transfers (2.82), the 
average for both the first and second se- 
mester cumulative GPA achieved at SWT 
was higher (ACC = 2.55, all two-year col- 
leges = 2.51). 

• Students transferring with either fewer than 
15 hours (2.81) or with greater than 60 
hours (2.73) achieved the highest GPA af- 
ter the first year at SWT. 

• By major, ACC students averaged a higher 
GPA in the areas of education (2.72), lib- 
eral arts (2.57), science (2.44), business 
(2.46) and fine arts/communications (2.64). 

• Students transferred mostly into the col- 
leges of Business (176), Liberal Arts (104), 
Fine Arts/Communications (100) and Edu- 
cation (85), with fewer students transfer- 
ring into Science (72), Applied Arts (54) 
and Health Professions (30). 

• When looking at the return rate for the sec- 
ond fall (2001), 80% of ACC students re- 
turned which is 2% higher than the return 
rate for all transfer students. 

— Christina Duncan 
Articulation Officer 
HBC 

'Decision GPA is the student’s cumulative GPA prior to being 
accepted at the university. The minimum must be at least 2,25. 



• Students transferred most into the colleges 
of Liberal Arts (169) and Natural Sciences 
(112), with the fewest number of students 
transferring into the college of Nursing (8), 
Social Work (6), Pharmacy (3) and Archi- 
tecture (2). 

• Regarding performance after two semesters 
at UT, ACC students averaged a higher 
GPA for courses taken in English, math, 
science, foreign language, social science 
and “other” areas when compared to the av- 
erage of all transfers from other Texas in- 
stitutions, with only a slight decrease 
(0.067) in the GPA of students taking vis- 
ual and performing arts. 
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Workforce Education Program Outcomes 



A critical component of institutional effec- 
tiveness is the assessment of student outcomes. 
For workforce programs, the Texas Higher Edu- 
cation Coordinating Board (THECB) has man- 
dated specific institutional effectiveness outcome 
measures for all community and technical col- 
leges. Two key measures for the quality of work- 
force education programs are the number of 
graduates produced over a three-year period and 
the successful placement of those graduates into 
the workforce or other institutions of higher edu- 
cation. 

The THECB reports data for these two meas- 
ures to community colleges annually. Each com- 
munity college must use these results to docu- 
ment the quality of programs when completing 
the Annual Institutional Self-Evaluation, which is 
a part of the Carl Perkins grant process each 
May. In addition, when a community college 
submits new program proposals, the college must 
also submit improvement strategies for any exist- 
ing program not currently meeting the standards 
for these two measures. The THECB’s logic be- 
hind this requirement is that new programs 
should not be approved if quality cannot be dem- 
onstrated in the existing programs. 

The THECB also requires a summary evalua- 
tion of the college once every four years. In the 
last year, the THECB has moved away from do- 
ing On-Site Visits to review institutional effec- 
tiveness and has created a Desk Review process. 
Currently, if a college is scheduled for review, 
the college may choose an On-Site Visit or a 
Desk Review. If the college does not meet stan- 
dards on the Desk Review, then an On-Site Visit 
is triggered. When a college has several work- 
force education programs that do not meet the 
quality standards for the number of graduates and 
for graduate placement, then a site visit will be 
initiated. (The site visit may be targeted to re- 
view just a few programs or it can be comprehen- 
sive.) 



Essentially, the THECB has provided these 
broad measures as minimum standards for work- 
force education programs because they serve as an 
efficient assessment of the program. If a program 
cannot produce sufficient graduates, this may indi- 
cate a lack of student demand, a lack of business/ 
industry demand, or even some curricular issues 
that need to be addressed. More importantly, if 
the graduates are not finding employment or oth- 
erwise continuing their education, it is worth ques- 
tioning whether the program is meeting its mis- 
sion. The measures stipulate the following: 

• Each workforce education program will 
produce at least 15 graduates over a three- 
year period, and 

• At least 85% of the graduates in each 
workforce education program will be 
placed within one year of graduation 
(successful placement is considered em- 
ployment or enrollment in a post- 
secondary educational institution and it is 
calculated over a three-year period). 

The three-year measurement period allows 
small or developing programs ample opportunity 
to balance graduate outcomes over a longer pe- 
riod. New programs are also given a three-year 
period to develop and produce graduates before 
their assessment begins. In the most recent report 
from the THECB, three new ACC programs 
(Biotechnology, Culinary Arts, and Pharmacy 
Technician) were not evaluated yet because they 
have not reached their three-year mark. 

ACC offers many different majors and spe- 
cializations across a variety of workforce educa- 
tion programs. The THECB evaluates a program 
as a whole (by the CIP area, in technical terms), so 
several individual majors or specializations may 
be included in the program figures. Again this 
helps to balance smaller specializations within a 
program area that may not be able to individually 
achieve the standards. 

(Continued on page 9) 
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Table 1 : THECB Workforce Education Outcomes 



CIP 


Workforce Program Area (THECB name) 


Total Over 3 Years (1996 - 1999) 


Program Meets 
Standard 


# Graduates 


Placement 
# % 


Graduates 


Placement 


11.0101 


Computer & Information Sciences, General 


30 


29 


96.7% 


y 


y 


11.0201 


Computer Programming 


24 


20 


83.3% 


y 


X 


15.0303 


Electrical, Electronic & Communications Engineer 


302 


292 


96.7% 


y 


y 


15.0501 


Heating, Air Conditioning & Refrigeration Technology 


24 


23 


95.8% 


y 




15.0702 


Quality Control Technology/Technician 


10 


10 


100.0% 


X 




15.1102 


Surveying 


13 


10 


76.9% 


X 


X 


20.0201 


Child Care & Guidance Workers & Managers, General 


44 


43 


97.7% 


y 


y 


22.0103 


Paralegal/Legal Assisting 


43 


41 


95.3% 


y 


y 


23.1101 


English Technical & Business Writing 


11 


11 


100.0% 


X 


y 


31.0501 


Health & Physical Education, General 


0 


0 


N/A 


X 


X 


43.01* 


Criminal Justice & Corrections 


162 


155 


95.7% 


y 




43.02* 


Fire Protection 


61 


61 


100.0% 


y 




46.0201 


Carpenter 


6 


6 


100.0% 


X 




47.0101 


Electrical & Electronics Equip. Installers & Repairer 


11 


11 


100.0% 


X 




47.06* 


Vehicle & Mobile Equipment Mechanics 


17 


12 


70.6% 




X 


47.0604 


Auto/Automotive Mechanic/Technician 


20 


20 


100.0% 




y 


48.0101 


Drafting, General 


200 


189 


94.5% 


y' 


y 


48.02* 


Graphic & Printing Equipment Operator 


28 


25 


89.3% 




y 


48.0508 


Welder/Welding Technologist 


45 


42 


93.3% 




y 


50.0402 


Graphic Design, Commercial Art & Illustration 


68 


59 


86.8% 






50.0406 


Commercial Photography 


24 


20 


83.3% 




X 


50.0909 


Music Business Management & Merchandising 


25 


23 


92.0% 






51 .0205 


Sign Language Interpreter 


10 


9 


90.0% 


X 




51.0803 


Occupational Therapy Assistant 


52 


48 


92.3% 


y 




51.0806 


Physical Therapy Assistant 


57 


51 


89.5% 




y 


51.09* 


Imaging Technology 


79 


77 


97.5% 






51 .0904 


Emergency Medical Technology/Technician 


23 


23 


100.0% 






51.0909 


Surgical/Operating Room Technician 


35 


33 


94.3% 






51.1004 


Medical Laboratory Technician 


31 


31 


100.0% 






51.15* 


Mental Health Services 


68 


62 


91.2% 






51.1601 


Nursing, General 


278 


272 


97.8% 




y 


51.1613 


Practical Nurse (LPN Training) 


130 


125 


96.2% 




y 


51.2309 


Recreational Therapy 


19 


17 


89.5% 


y 


y 


52.02* 


Business Administration & Management 


117 


102 


87.2% 


y 


y 


52.0302 


Accounting Technician 


39 


38 


97.4% 




y 


52.04* 


Administrative & Secretarial Services 


55 


52 


94.5% 




y 


52.08* 


Financial Management & Services 


13 


13 


100.0% 


X 


y 


52.09* 


Hospitality Services Management 


8 


7 


87.5% 


X 




52.1204 


Business Systems Networking & Telecommunications 


31 


27 


87.1% 


y 




52.1401 


Business Marketing & Marketing Management 


11 


9 


81.8% 


X 


X 


52.1501 


Real Estate 


2 2 


100.0% 


X 





Note: New programs and Programs that were closed have been excluded from the report. 

* Combined programs with two or more individual award areas. THECB evaluates these programs as a single group. 
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(Continued from page 7) 

A review of the table (page 8) shows most 
of ACC’s workforce education programs are in 
fine shape (note the program areas are identi- 
fied by the program inventory title used by 
THECB, not by the local ACC name). The 
programs with large numbers of graduates are 
doing well with their placement rates. Five out 
of the six programs with more than 100 gradu- 
ates had close to or over a 95% placement rate. 
These largest programs include Electronics 
Technology, Nursing (Professional and Voca- 
tional), Engineering Design Graphics, Crimi- 
nal Justice, and Business Administration. 

Overall, there were eleven program areas 
that did not meet the fifteen graduate standard. 
These programs tend to have relatively small 
enrollments and the number of graduates can 
fluctuate from year to year. The programs not 
meeting the graduate standard include Quality 
Assurance, Geomatic/Land Surveying, Techni- 
cal Communications, Physical Fitness Tech- 
nology, Building Construction Technology, 
Consumer Electronics Repair, Interpreter 
Preparation, Financial Management, Hospital- 
ity Management, Marketing, and Real Estate. 

The majority of workforce programs met 
the placement standard for their graduates. 
The standard is 85% placement. The THECB 
identifies placement as employment or attend- 
ing a post-secondary institution. Therefore, 
students who are still enrolled in college after 
graduation will be counted as successful even 
if they are not working. The presumption is 
that the student is continuing his or her educa- 
tion to meet long term employment objectives. 
The placement rate includes all students that 
were found through the follow-up process. 
(See “How the THECB Gets the Data,” at the 
end of this article.) Some students will be in- 
dicated as “not found” since they have moved 
out of state or otherwise cannot be located on 
state employment databases. 
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The six programs that did not meet this stan- 
dard tended to have 20 or fewer graduates, 
which means that one missing graduate could 
influence the results by 5% or more. The six 
programs not meeting the placement standard 
are Computer Programming (only one part of 
the Computer Information Technology pro- 
gram), Geomatics/Land Surveying, Physical Fit- 
ness Technology, the Vehicle and Mobile 
Equipment Mechanics area (the certificate pro- 
grams in Automotive Technology - Small En- 
gine, Marine Engine, and Motorcycle Repair), 
Photographic Technology, and Marketing. 

How the THECB Gets the Data 

Note: This is a simplified presentation of a complex 
process, which actually involves the Texas Worlrforce 
Commission as the active agent for several steps. 

Each year the Texas Higher Education Coor- 
dinating Board (THECB) produces a series of 
reports on the outcomes of community college 
graduates and leavers (students who left the in- 
stitution without completing a program). The 
follow-up process conducted by the state is 
called the Automated Student and Adult Learner 
Follow-up System (ASALFS). It includes the 
following steps: 

1. Each fall, Austin Community College 
submits the CBM009 Graduate Report to 
the THECB for students that have gradu- 
ated the previous academic year 
(December, May, and August). 

2. THECB then checks whether those stu- 
dents are enrolled in a post-secondary 
institution in Texas in the fall semester 
following their graduation to determine 
if they are pursuing additional education. 

3. THECB identifies employment in the 
year following graduation with the Un- 
employment Insurance Wage Records 
that most businesses submit to the state. 
(Some smaller businesses do not submit 
these tax records to the state.) 

(Continued on page 10) 
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4. The data on employment and additional 
education are combined with the original 
graduate report to produce the initial place- 
ment report. 

5. The student records that were not identi- 
fied by the state follow-up are sent to the 
College for the Supplementary Follow-up 
(on a report called CBM 1 16). 

6. With the assistance of faculty and staff at 
the program level, the missing students are 
located, and their status is then reported 
back to the THECB. 

The entire process is completed one and a half 
years after the College submits the original gradu- 
ate report. For example, ACC submitted to the 



THECB its graduate report for the 1999-2000 aca- 
demic year in November 2000. The Supplemen- 
tary Follow-up report will be due to the THECB 
by March 2002 and the entire process with com- 
plete data will be finished by April 2002. THECB 
collects and reports additional data regarding the 
training-relatedness of the student’s occupation 
and wage information on a limited number of 
graduates, but this report has a lag time of slightly 
over two years. 

— Christopher Vinger, 
Coordinator, Analysis 
Office of Institutional Effectiveness 

HBC 



“You Can Learn, but You Can’t Park” 



Introduction 

In Spring 2001, Austin Community College 
participated with twelve south Texas community 
colleges in a student satisfaction survey devel- 
oped by representatives of the Gulf Coast Consor- 
tium of Institutional Research (GCAIR) in re- 
sponse to the need for data that could be used to 
compare the perceptions and preferences of stu- 
dents across institutions. Nearly 500 surveys from 
ACC students were collected during the study (N 
= 497). 

This was the first year that a college outside 
of the Gulf Coast Consortium has participated. A 
major purpose of this survey was to generate 
benchmarks for the items dealing with student 
satisfaction with college services. Participating 
colleges could, thus, determine whether their stu- 
dents were more or less satisfied with these ser- 
vices than students at similar institutions. 

The survey focused on reasons why students 
chose the attended college, perceptions of extra- 
curricular activities, importance of and satisfac- 
tion with college services, and students’ percep- 
tions of the college in general. 



The Institutional Research Office at San Ja- 
cinto Community College collected and summa- 
rized the data into tables and distributed them to 
each of the other participating colleges. ACC ex- 
cerpted its data and used it to compare its analysis 
of student survey responses with those of three 
other colleges (comparable in enrollment): Hous- 
ton Community College, North Harris Montgom- 
ery Community College, and San Jacinto Commu- 
nity College. 

Findings 

Demographics & Miscellaneous 

A demographic analysis of the ACC students 
that participated in the 2001 GCAIR survey re- 
vealed that, overall, the sample was representative 
of the ACC student body. ACC students participat- 
ing in the 2001 GCC survey were comparable to 
the ACC student body population in gender, age, 
and ethnicity. 

• Over 75% (77.3%) of ACC respondents 
indicated that they worked either full-time 
or part-time off campus. 

• The majority of ACC students (62.3%) in- 
dicated that their main educational goal 
was to transfer to a 4-year institution. 

(Continued on page 11) 
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(Continued from page 10) 

• Nearly 40% of ACC students surveyed indi- 
cated that they would like to transfer to the 
University of Texas (38.7%) and another 
32% (31.9%) reported Southwest Texas 
State as their transfer destination. 

Note: In the analysis that follows, averages are 

given for male respondents (M) and for female re- 
spondents (F) separately. 

Why Students Chose College 

Course-related items were cited by ACC survey 
respondents as important factors in their decisions 
to attend ACC. The ranking system used 1 = “Not 
a reason to attend,” 2 = “Minor reason to attend- 
ing,” and 3 = “Major reason for attending.” ACC 
survey respondents cited as the two most important 
factors they considered when selecting a college, 
“Offered courses I wanted” (M avg. 2.66 and F 
avg. 2.73) and “Offered courses I wanted at times I 
wanted” (M avg. 2.51 and F avg. 2.79). Analysis of 
the 3 comparison colleges revealed that, like ACC 
survey respondents, students selected “Offered 
courses I wanted” and “Offered courses I wanted at 
times I wanted” as important factors in their deci- 
sion regarding which college to attend. 

In addition, ACC survey respondents deemed a 
variety of other items as important in their decision 
to attend ACC. These included the following: 

• “Low cost of tuition and fees” (M avg. 2.44 
and F avg. 2.61), 

• “Convenient location” (M avg. 2.44 and F 
avg. 2.49), 

• “Work while attending” (M avg. 2.45 and F 
avg. 2.44), and 

• “Good chance of personal success” (M avg. 
2.32 and F avg. 2.42). 

ACC survey respondents did not rate “Advice 
of High School counselor/teacher” (M avg. 1.23 



and F avg. 1 .3 1) as very important. ACC survey 
respondents had the lowest importance rating of 
all participating colleges on this item. 

Extra-Curricular Activities 

Of respondents who marked “Yes, fre- 
quently” to the items listed under participation in 
extra-curricular activities, male ACC survey re- 
spondents reported high participation rates in 
athletic activities with “Fitness Center” (1 1.7% ) 
and “Use tennis courts, running track, 
etc.” (7.7%) as the activities most frequently 
utilized. Female ACC survey participants re- 
ported the “Fitness center” (6.8%) and “Visit art 
gallery” (3.9%) as frequent extracurricular ac- 
tivities. 

Both male and female ACC survey respon- 
dents indicated low participation in “Student 
government” (with 3.6% of males and 1.8% of 
females responding “Yes, frequently”) and 
“College sponsored dances/parties” (2.6% of 
males and 1 . 1% of females responding “Yes, fre- 
quently”), activities also reported to be infre- 
quently used by survey respondents from all col- 
leges. 

Importance of College Services 

The college services deemed most important 
by ACC survey respondents were consistent 
with their reasons for choosing the college. On a 
5-point scale, where 1 = “Not important at all” 
and 5 = “Very important,” ACC male and fe- 
male respondents reported “Availability of 
courses at times needed” (M avg. 4.46 and F 
avg. 4.70) and “Variety of courses” (M avg. 
4.42 and F avg. 4.56) as important college ser- 
vices. 

Interestingly, female ACC survey respon- 
dents rated college services that dealt with fac- 
ulty/staff-student relations as more important 
than did their male counterparts. Specifically, 
female ACC survey respondents reported higher 
levels of importance for the following: 

• “Racial harmony at this college” (M avg. 

3 .66 and F avg. 4.16), 
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• “Attitude of non-teaching staff to stu- 
dents” (M avg. 3.79 and F avg. 4.21), 

• “Attitude of faculty/staff to students of 
your ethnicity” (M avg. 3.64 and F avg. 
4.22), 

• “Attitude of faculty/staff to students taking 
developmental courses” (M avg. 3.59 and 
F avg. 4.10), 

• “Attitude of faculty/staff to disabled stu- 
dents” (M avg. 3.71 and F avg. 4.22), and 

• “Attitude of faculty/staff to students of 
your gender” (M avg. 3.55 and F avg. 
4.21). 

Similarly, female ACC survey respondents 
placed higher importance on “Financial aid ser- 
vices” (M avg. 3.70 and 4.16) than their male 
ACC counterparts. 

College services considered less important to 
ACC survey respondents, as well as to all survey 
respondents, were “Student govt. /senate/ 
congress” (M avg. 2.88 and 2.99) and “ Varsity 
athletics” (M avg. 2.91 and 2.54). 

Satisfaction with College Services 

ACC survey respondents were also asked to 
rate their satisfaction on college services they pre- 
viously rated on importance (1 = “Very dissatis- 
fied”; 5 = “Very satisfied”). Both male and female 
ACC survey respondents were satisfied with 
“Telephone registration” at the college (M avg. 
3.98 and F avg. 4.14) and “Availability of senior 
institution transfer plans” (M avg. 3.94 and F avg. 
3.95). 

Female ACC survey respondents were more 
satisfied with “Information provided in course 
schedule” (M avg. 3.80 and F avg. 4.08) than their 
male counterparts. 



As with the importance scores, female ACC 
survey respondents reported higher satisfaction 
ratings than males for faculty/staff-student rela- 
tions: 

• “Racial harmony at this college” (M avg. 
3.78 and F avg. 3.83), 

• “Attitude of faculty/staff to students of 
your ethnicity” (M avg. 3.74 and F avg. 
3.94), 

• “Attitude of faculty/staff to students tak- 
ing developmental courses” (M avg. 3.62 
and 3.90), 

• “Attitude of faculty/staff to disabled stu- 
dents” (M avg. 3.71 and F avg. 3.87), and 

• “Attitude of faculty/staff to students of 
your gender” (M avg. 3.68 and F avg. 
3.93). 

Comparison of Satisfaction and Importance of 
College Services 

Further analysis of ACC survey respondents’ 
perceptions of college services revealed a decid- 
edly large gap between satisfaction and impor- 
tance ratings for “Parking facilities and ser- 
vices” (M avg. -1.07 and F avg. -1.22). This 
negative difference score indicates that ACC sur- 
vey respondents place a high importance level on 
parking at ACC, yet are not very satisfied. The 
“high-importance - low-satisfaction” finding for 
parking facilities is a finding that is common 
across the majority of colleges participating in 
this survey. 

Other negative gap scores for ACC survey 
respondents were found in course-related ser- 
vices: “Availability of courses at times 

needed” (M avg. -.62 and F avg. -.78) and 
“Variety of courses” (M avg. -.52 and F avg. - 
.56), an important finding given the value of 
these services to ACC respondents. 

(Continued on page 13) 
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Perceptions of the College in General 

The majority of ACC students surveyed ex- 
pressed satisfaction with the college and the educa- 
tion that they were receiving. However, male ACC 
survey respondents were less likely than female 
students to indicate that they would choose the 
same college if they could start again (70.3% of 
males vs. 81.9% of females responding “Definitely 
or probably yes to attending this college if you 
could start college over”) and less likely to describe 
the quality of the education that they received as 
“good” or “excellent” (70.6% of males vs. 80.2% 
of females). This trend is also evident at the other 
three colleges. 

Summary 

The GCAIR survey was a very valuable tool for 
analyzing ACC students’ perceptions on a variety 
of issues and for making comparisons with other, 
similar community colleges. 



An important finding in the report was the high 
importance placed by ACC students on variety and 
availability of courses and the corresponding gap 
in satisfaction with these services. This finding, 
combined with the finding that course offerings 
(variety and time) were major reasons for attending 
the institution, indicates that this is an issue that 
may need to be addressed. 

Cost and location, always important to the stu- 
dent population, were also important reasons stu- 
dents gave for attending ACC. 

And finally, the gender gap between male and 
female ratings of items dealing with faculty/staff- 
student relations, needs to be examined further. 

— Z\y Shafir, 
Coordinator, Analysis 
Office of Institutional Effectiveness 

HBC 




General Education Assessment 

http://www.austin.cc.tx.us/gened/ 

Workforce Education 

http://www.thecb.state.tx.us/ctc/ie/ 

Transfer Programs 

http://www2.austin.cc.tx.us./oiepub/initiatives/assessment/measures.html 

and 

http://www2.austin.cc.tx.us./transfer 

Gulf Coast Consortium of Institutional Research 
2001 Survey Results 

http://www.sjcd.cc.tx.us/research/gcair01.pdf 

and 

http ://www2.austin.cc.tx.us./oiepub/pubs/gcc_2001 spring.pdf 

Community College Survey of Student Engagement 

http://www.ccsse.org/survey/survey2.pdf 
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...Getting Better all the Time 



Austin Community College’s Internal Cus- 
tomer Satisfaction Survey (ICSS) conducted in the 
spring of 2001, identified fourteen “highly-used” 
offices with twenty percent or more respondents 
reporting they were dissatisfied with the office’s 
overall service. These offices included the follow- 
ing: 

ACCNet Services 
Accounts Payable 
Compensation/Classification 
Continuing Education 
Employee Benefits 
Employment/Hiring Services 
Faculty Evaluation 
HBC Room Scheduling 
Marketing & Public Information 
Office of the AVP Human Resources 
Office of the Dean of Arts & Humanities 
Office of the Dean of Math & Sciences 
Payroll Office 
Personnel Records 

In September, 2001, each of these offices 
was asked to report on the status of the actions 
they planned to take (or had already taken) to im- 
prove the quality of their overall service, the 
promptness of their service, and their service atti- 
tude. 

Implementation of Improvements 

An examination of all of the status reports 
submitted revealed a total of 100 improvements 
planned by the fourteen Highly Used/High Dissat- 
isfaction (HU/HD) offices in response to the ICSS 
conducted this past spring. As of October 1, 2001, 
twenty-nine of these had been completed and 
thirty-seven are expected to be completed within 
the next six months. Of those remaining, nineteen 
are “on-going,” and the rest are scheduled to be 
completed by December 2002. 

Types of Improvements 

The status reports also revealed noticeable 
similarities in the actions the fourteen Highly 
Used/High Dissatisfaction offices planned to take 



to improve their services, as well as improvements 
unique to the function each office fulfills. The 
similarities in improvement actions can be catego- 
rized in the following ways: 

• Hire, upgrade, or reassign staff. Eleven (79%) 
of the (HU/HD) offices planned to do this. 

• Develop on-line or software applications. 
Eleven (79%) of the (HU/HD) offices planned 
to do this. 

• Participate in customer service training. Ten 
(71%) of the (HU/HD) offices planned to do 
this. 

• Establish or write procedures for customers or 
staff. Nine (64%) of the (HU/HD) offices 
planned to do this. 

• Increase teamwork within the office by cross 
training, staff meetings, staff retreat, or use of 
external consultants. Nine (64%) of (HU/HD) 
offices planned to do this. 

• Take actions to improve communication both 
within the office and with customers. Six 
(43%) of (HU/HD) offices planned to do this. 

• Collect customer feedback via point-of-service 
survey, “card,” or focus groups. Six (43% of 
(HU/HD) offices planned to do this. 

• Train or retrain staff. Four (29%) of (HU/HD) 
offices planned to do this. 

It will be interesting to see the impact of these im- 
provements on the results of the Internal Customer 
Survey of Spring 2002. 

— Roslyn Wallace , Coordinator, 
Institutional Assessment 
Office of Institutional Effectiveness 

HBC 



"AGE 14 



100 



ACC Effectiveness Revie 7 v spring 2002 



ACC Effectiveness Review 




Roslyn Wallace, Coordinator of Institutional Assessment, 
answers questions about creating and documenting assess- 
ment plans and activities. You may contact her by email 
(rwall@austin.cc.tx.us) or by phone (223-7585). 



Q_: I’ve heard that because we are an academic/ 
transfer discipline, we are required to measure the 
percent and performance of students that transfer 
to a four-year school to demonstrate that our unit 
is effective. There are two problems with this. 
First, students do not “major” in our discipline, 
but take courses because it fulfills the requirement 
for an elective for their major. Second, we do not 
have access to information on which of our stu- 
dents continued their study of our discipline at any 
specific four-year institution, thus we also do not 
know how well they did. Can we use something 
else to measure the effectiveness of our program? 

Sure. There is no requirement that you 

measure any particular outcome to demonstrate 
the effectiveness of your unit. The goal of con- 
ducting an assessment is to provide information 
you can use to improve your students learning. 

A quick review of the assessment plans in the 
U-LEAD reveals several methods academic units 
use to discover areas in which improvements to 
student learning may occur. 

•The art department invites university faculty to 
judge students’ work based on agreed upon crite- 
ria. 

•Many academic disciplines measure student 
mastery of specific course content to discover 
weaknesses in the curriculum so that improve- 
ments may be implemented. For example, the bi- 
ology department looks at the specific competen- 
cies students should have mastered in the 
“foundation” course if they are to be successful in 
the “target” course, either within their discipline 
or other disciplines. Developmental writing ex- 
amines their students’ performance in English 
composition I. 

•Other disciplines examine grade distributions 
across two courses to gather information on how 
well students are learning. Chemistry, for exam- 



ple looks at grade distributions for general chem- 
istry I and general chemistry II. 

•The government department administers an 
“exit” exam to all sections of GOVT 2306 to ex- 
amine student performance. 

•The speech department relies on standards es- 
tablished by the National Communication Asso- 
ciation to assist them in examining students’ oral 
competency skills. 

In disciplines that are science related, experi- 
ences that measure students’ mastery of the scien- 
tific method could be useful in providing feedback 
to the discipline. Another method of gathering 
feedback to determine the effectiveness of a 
course could include measuring change in stu- 
dents’ attitudes. In environmental science, for ex- 
ample, the difference in the number of students 
who recycle at the end of the course and the num- 
ber who did so at the beginning of the course 
might indicate how well the course influenced stu- 
dent behavior. 

The important thing is to design your assess- 
ment so that it provides information you can use 
to improve your students’ learning. The only re- 
quirement is that you assess your course/ 
program’s effectiveness each academic year 
(however you choose to do that) and use the re- 
sults to make improvements. 

Q_: I got a memo the other day that said assess- 
ment plans had to be documented into the Unit- 
Level Effectiveness Assessment Database (U- 
LEAD) by now. I thought the deadline for docu- 
menting these was not until June 15. Which is 
true? 

It is true that the deadline for documenting 

all of your unit’s assessment activities into the U- 
LEAD is June 15. The memo was a reminder that 
the outcome, criteria, and methodology for this 
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academic year (the assessment plan) should be 
documented into the database by now. Later, by 
June 15, the unit will document the results of the 
assessment plan and the improvement actions it 
plans to implement. Of course you may wait until 
June 15 to complete all your documentation, in- 
cluding the assessment plan. 

Qj I know I typed my unit’s assessment plan for 
this year into the U-LEAD and hit the submit but- 
ton. When I went back to add the results, I could 
not find it. Where did it go? Did someone else 
delete it? Do I need to re-do all that work, or is 
there some way I can find it? 

Not to worry! Contact me. I can find the as- 
sessment plan in the database. Only the database 
administrator or I can delete records from the da- 
tabase. You may have forgotten to select the unit 
name on the record. 

Cb I was trying to document my assessment 
plans for this year into the U-LEAD, but couldn’t 
submit the new record because my password 
wouldn’t work. How do I “get in?” 

c/t The user name and password for the U- 
LEAD is the same for everyone (user name: ACC 
Units, password: unitgoals); your email password 
and username won’t work. 



Cb What do we do if there have been changes in 

our unit and the assessment plans that are in the 
database are no longer appropriate or we do not 
have access to any of the data that were supposed 
to have been collected? 

(/t I presume we are talking about assessment 

plans from a previous year. If so, there’s not 
much you can do. If you have exhausted all pos- 
sibilities for obtaining the information, I suggest 
you indicate on the original assessment plan that 
the assessment was not done and give the reason. 

Qj Someone told me that we get professional de- 
velopment credit for documenting our unit’s as- 
sessment activities into the U-LEAD. Is that true? 

(/t Not exactly... but you can earn two hours of 
professional development credit if you complete 
the assessment workshop that is designed to assist 
you in creating and documenting unit-level as- 
sessment plans. There is one offered for instruc- 
tional units on the first Friday of every month, 
one month for instructional units and the next 
month for non-instructional units. Faculty may 
sign up through Faculty Development (http://irt. 
austin.cc.tx.us/registration/): staff need to call me. 

rfl: The Unit-Level Effectiveness Assessment 
Database is being redesigned. Documenting as- 
sessment activities should be easier under the new 
design. New data entry manuals will be distrib- 
uted before the changeover occurs this spring. 
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